Generally, mammary glands are located in several regions. They can be located anteriorly (as in primates, elephants, sea cows and bats), posteriorly (as in horses, cows, sheep, and whales) or serially (as in litter bearing species such as dogs and cats). Anterior mamary glands are located in the thoracic region, whereas posterior mammary glands are located in the inguinal region.
In addition to supplying young Solenodon with nourishment, the teats provide a secondary function. Namely, they aid in the transport of solendon young. From an intersting article on Solenodons:
We observed a unique mode of maternal-young contact which we have referred to as 'teat transport'. This phenomenon is well known among some rodents (2), but unreported in insectivores. At seven weeks of age the youngster will accompany its mother on her foraging activities by clinging to one of the two inguinal teats. At this time the teats are very elongated, up to 2 cm in length, enabling the youngster to cling to a teat as it is dragged along close behind. As the infant grows it is able to assert its own locomotion and simply seize the teat and follow, moving when the mother moves, stopping when she stops. The 21/2 month old infant may still show this response and even scratch itself while standing behind the mother holding on to a teat. It would seem that if the solenodon has to change burrows from time to time, then such a teat-transport mechanism enables the female to move with still very dependent young, pulling them along behind her on her teats, rather than attempting to carry them in her mouth. Mouth transport, of course, is a wide-spread phenomenon in small insectivores; however, the solenodon's teat-transport mechanism is probably quite efficient since the young remain dependent for a long time. During this dependency period, the female can forage and be accompanied by the young.
This is similar to the function of inguinal teats in marsupials - the one difference being that in marsupials the teats used for transport do not produce milk. I could find no information on the actual anatomy of the Solenodon mammary gland so I can't compare it to, say, marsupials, cows, sheep, or humans. Although, since Solendon are pretty ancient (as a species - dating back to the ime of the dinosaurs) I would expect there to be some similarities with marsupials.
I also wonder if the teat transport might not explain why there are so few venomous mammals - most mammals transport their young by carrying them with their teeth. I have no data on the subject so this is pure speculation. One of the reasons Solenodon was free to evolve venom is that they used teat transport to move their young and wouldn't run the risk of accidently poisoning their young (incidently, nursing is one way mammals build up immunity so a possible test of this might be to examine young Solenodon's immunity to their own poison). Other mammals, then would have been precluded from developing venom because an alternative transport mechanism was not avialable for co-option. If I undestand Gould correctly, this would be a case of exaption.
An intersting link Comparitive Mammary Gland Anatomy