According to the social security website, you automatically get Parts A and B after you have been receiving SS disability benefits for 24 months. HOWEVER, you said you gave it up years ago. The problem may be that since you gave it up voluntarily, you may be required to pay a penalty if you try to sign up now.
The penalty for late signup for Part B is 10% on the premium per year for each year you were eligible for Part B but failed to sign up for it. So, if you waited 4 years, your increase in premium would be 40%.
HOWEVER, when you turn 65, this may change. The AARP website posts this question and answer by one of their Medicare consultants, which seems to mirror your dilemma:
"Q. My wife went on Medicare because of disability in 1995, but later dropped Part B because she was covered on my employer’s health plan. When I lost my insurance, she went back on Part B. But she has to pay a large late penalty on top of her premiums. She’s currently 63. Is there any way out of this situation?
A. Yes. When your wife turns 65, she’ll no longer pay the Part B late penalty. Here’s why:
When you qualify for Social Security disability payments under age 65, after a time you also become entitled to Medicare health coverage. You receive Medicare Part A (hospital services) automatically and can choose to enroll in Part B (doctors and other outpatient services) within a certain time frame. This time frame counts as your “initial enrollment period.” If you don’t sign up for Part B during this time, or enroll and drop out later on, you’ll incur a late penalty if you do finally enroll in Part B while still under age 65. The late penalty is an additional 10 percent of the Part B premium for each full year that you were without Part B when you were eligible for it.
But everything changes when you reach 65. At the end of the month before the month in which you turn 65, you lose your entitlement to Medicare based on disability. At the beginning of the month you turn 65, your entitlement to Medicare based on becoming 65 begins. In other words, you get a second initial enrollment period. At that point the clock is reset, and Medicare coverage begins anew as though you’d never had it before.
This arrangement means that if you were paying any late penalties for Part B or Part D (drug coverage), you will no longer do so."
SO, PLEASE keep after Social Security (I have better luck on their 800 number than I do with our local office, even in person). Our local office here does not seem always to give out good information for whatever reason. As someone above said, you can call the 800 number and make an appointment for them to call you back, and they actually DO. Get the number from medicare.gov website.
Be sure you tell them all your details, about your age and about the fact that you turned down part B when you did because you couldn't afford it. According to what I am reading on the Medicare website, I think you can get around the penalty if you are not yet 65, and probably even if you are 65 the penalty would run from when you turned 65 and not when you turned Part B down. Stay on them until you get a clear answer and then ask them to confirm it in writing. If the first agent you get can't help you, ask for a supervisor. Be prepared to spend a while on the phone. You have to be your own advocate, and stay after them (politely, of course, until you get a definitive answer).
ALSO, please know that there is a Medicare program called "Extra Help" which helps pay your deductible for Part D prescriptions; it is based on income. You have to apply for it and the people on the phone can tell you how. We were able to get this for my sister, and it makes a big difference for her, especially on a couple of drugs she takes which are Tier 3 drugs. Please don't decline Part D coverage -- you will seriously regret it later on.