Exactly because it was and still is rampant.
I understand that what you have in mind is rich, fat, Greeks driving around in Porsches and Ferraris because of not paying taxes but that is not the case.
The problem of tax evasion in Greece, one that the IMF has repeatedly called is that Greece has a small tax base. A lot of people do not pay taxes because most of them are struggling to make ends meet. Greece like most of south Europe in the past has a large number of family businesses and a large number of freelancers. In fact most lawyers, most engineers and even most doctors do not work for large companies but are self employed. As entities they are “small fish” and each mom&pop shop or freelancer does not cost the state a lot of revenues when s/he doesn’t pay taxes. However, all of the lost revenue totaled really adds up.
The next problem is that it is next to impossible to check the books of each and every one of them. The Laws are in place and they are not lax at all. And although governments do not prosecute people, judges do. And some people do end up in jails.
So? Greece has an inmate capacity of 6000 and currently has more than 11000 prisoners (numbers may be a little dated). If a court incarcerates the 70-year old grocer for not giving a receipt, who is it going to release? And let’s say Greece builds more prisons or even goes all demented and allows private prisons… Then what? Who will pay for the incarceration of the old guy? And how will his incarceration earn money for the state? By example you say? OK, then who’s going to make all the checks for the period of many years that will be needed for the example to be understood by all citizens of Greece? More employees hired at the Greek IRS? More civil servants? Or maybe privatise that too?
“Halt! This is the new IRS! Do you have an invoice for all the products your donkey is carrying? Hm… No invoice, no shipping bill, no manifest, right. Sorry uncle Tom, you’ re going straight to jail. Don’t worry you can pick cotton in our prison’s fields to cover some of your costs.”
And you know what makes me truly angry with this line of reasoning?
That entities like the IMF and the financial think-tanks do not “advise” Greece for the sake of Greeks. The real solution to all of Greece’s tax evasion problem according to them is pretty simple actually:
Greece and any country, should NOT have freelancers. It should have companies and corporations that have employees. Greece and any country should NOT have mom&pop stores. It should have retail chains and franchises.
This way everyone will be obliged to pay taxes.
If some company doesn’t, it will be easy to catch in an audit. Plus there can’t be hundreds of thousands of big companies with many employees all in need of an audit.
And it sounds just about right…
…until the moment you actually check how much tax big corporations pay. Or how many ways to dodge paying tax there are in the US, not to mention all the tax havens in the world.
Or how much pressure they can apply to the state to AVOID paying taxes!    (I could go on with the footnotes, but you get the point). Examples are from the US because you used it in the question. The same story applies to most countries one way or another.
Bonus material: You can’t really imprison a company. Not even when gross negligence and fraud are the least of its crimes!