Delete command doesn’t remove data !

[humananimal]

Reminder -
a ‘delete’ command or a ‘quick format’ doesn’t remove any data!
It just make it ready to be used again.
Like wiping the directory in a book, but the text is untouched.
If not totally rewritten or fully formatted, then a simple un-delete program can find the files again.

 

[pes]

This is actually a widely known piece of information.

And it is even worse than you think.
Traditionally on magnetic disks this was solved by rewriting the sectors or the entire disk with random data multiple times. That would sufficiently erase the data.
On SSDs and drives that use wear protection though because of the wear distribution firmware there are multiple copies of variously old versions of the same file and not only rewriting file with something like shred -f does not delete all of it, it damages the disk and may leave up to 50% of the old copies intact. Read the research on it.

 

[humananimal]

This is actually a widely known piece of information.

And it is even worse than you think.
Traditionally on magnetic disks this was solved by rewriting the sectors or the entire disk with random data multiple times. That would sufficiently erase the data.
On SSDs and drives that use wear protection though because of the wear distribution firmware there are multiple copies of variously old versions of the same file and not only rewriting file with something like shred -f does not delete all of it, it damages the disk and may leave up to 50% of the old copies intact. Read the research on it.

It widely known among people with some knowledge of the function of computers. But I haven’t meet ‘non-tech’ users that was aware of this. They are all sure that when pressed ‘delete’ it is gone forever. And even worse, many don’t know about the computers / phones / tablets ‘garbage bin’.
I was mostly thinking of the function on magnetic discs.
It is knew to me that there is a row of older files on SSD, and the word ’wear distribution firmware’ too. I guess it is an inbuilt function on SSD’s. I can see the point in using it on USBdrives, so the original file is still there if the stick is pulled out before all data is written. But why on SSDs?
- So it seems to me that the only solution is a full formatting to get rid of it all.
- Do you know what happens when using the inbuilt function on phones and iPads to clear itself before selling it ? Is all the data still there? And more important, can it be extracted by software?
- There is programs to over-write unused part of the ‘disc’ (I have used KillDisk on my PC). Do you know if they will overwrite such ‘old versions’ ?

 

[pes]

I can see the point in using it on USBdrives, so the original file is still there if the stick is pulled out before all data is written. But why on SSDs?

The wear distribution algorithm is not protecting data, it is protecting the drive. SSD and in general flash drives have a limited number of writes. Then the flash cells get damaged. So these algorithms spread the write instructions across the whole cell array to distribute writes and not overuse one portion of the disk. This extends the life of the drive. But it leaves copies of previously written data in those places since the algorithm does not in general keep track of the older copies, only the last one.

And even worse, many don’t know about the computers / phones / tablets ‘garbage bin’.

Do not even get me started on the """security""" of mobile devices.

- So it seems to me that the only solution is a full formatting to get rid of it all.

Might be. But that way you are rewriting all the cells every time which might significantly reduce the lifespan of the drive. And this is impossible to do on a drive that is used for something else also, like the internal storage of a phone without damaging the rest of the system.

- Do you know what happens when using the inbuilt function on phones and iPads to clear itself before selling it ? Is all the data still there? And more important, can it be extracted by software?

Nowadays phones are usually encrypted. This function destroys the encryption header making the rest of the drive practically impossible to decrypt and recover. It is usually not rewriting the whole disk.
But this is obviously unusable if you want to delete just a couple of files.

- There is programs to over-write unused part of the ‘disc’ (I have used KillDisk on my PC). Do you know if they will overwrite such ‘old versions’ ?

If you are using an SSD the success rates are different. Google the research about it.

 

[humananimal]

Thanks. Very informative.

 

[pes]

https://www.usenix.org/legacy/events/fast11/tech/full_papers/Wei.pdf

In other words do not save porn, do not make porn if that is a problem by the laws of your country, and if you do, use magnetic drives.
For anything else use an amnesic system such as tails linux which runs from RAM. And switch the computer off for at least 5 minutes when you are done. RAM is volatile memory, without power it looses data usually in 2 minutes.

And if your threat model is the government, place your computer in a box with about 50kg of TNT and never use the internet.

 

[humananimal]

https://www.usenix.org/legacy/events/fast11/tech/full_papers/Wei.pdf

In other words do not save porn, do not make porn if that is a problem by the laws of your country, and if you do, use magnetic drives.
For anything else use an amnesic system such as tails linux which runs from RAM. And switch the computer off for at least 5 minutes when you are done. RAM is volatile memory, without power it looses data usually in 2 minutes.

And if your threat model is the government, place your computer in a box with about 50kg of TNT and never use the internet.

I have read, well skimmed through, this paper. So until the manufacturers follow their recommendations, the only reliable way is to cling to the good old (and surprisingly reliable) magnetic HardDrives. I will get a couple from my drawer to use at my PC.

So as it is now - all our phones and tablets, and also PC’s under 5 years of age can’t erase single files in a secure manner.
But what about encryption? not even this works when some authority succeeds to remove the memory and performs a hardware-read. Because the password is probably readable on the SSD or Flash.
Most private phones back up ’in the cloud’. And if the cloud also uses SSD’s, then a ’delete’ will probably not work either.

And we were told that the new technology would make everything easier - oh dear

 

[pes]

But what about encryption? not even this works when some authority succeeds to remove the memory and performs a hardware-read. Because the password is probably readable on the SSD or Flash.

That would not make sense. The password can not be stored on the disk itself because that defeats the purpose of encryption. I do not know how exactly disk encryption work nowadays but in general you have a one way mathematical function that converts the password only you know into something you can compare to the encryption header and if that is the same, the disk opens. But it is impossible or very difficult to come up with the correct password if you have just the header. So without the password you can not decrypt the drive even if you do have low level access to the hardware.

One possibility to erase flash drives could be to encrypt them and then destroy the encryption. The data would remain but unreadable. But this is just an idea, I did not test it.

Most private phones back up ’in the cloud’. And if the cloud also uses SSD’s, then a ’delete’ will probably not work either.

Cloud backup is the least safe option there is. No matter what storage they use. Because the laws in the place where the cloud is are used for that data and no company ever is going to take the hit for you if they get into trouble.
Do not use that at all.

 

[humananimal]

That would not make sense. The password can not be stored on the disk itself because that defeats the purpose of encryption. I do not know how exactly disk encryption work nowadays but in general you have a one way mathematical function that converts the password only you know into something you can compare to the encryption header and if that is the same, the disk opens. But it is impossible or very difficult to come up with the correct password if you have just the header. So without the password you can not decrypt the drive even if you do have low level access to the hardware.

When using an external SSD or Flash drive there is no need to store the password on this disc. But locally on my PC everything has to be stored on the same disc. I think that the password to access my phone or tablet is not only in my brain, but also somewhere on the phone.

I wrote about this, referring to section 3.2.4 Encryption :
“Many recently-introduced SSDs encrypt data by default, because it provides increased security. It also provides a quick means to sanitize the device, since deleting the encryption key will, in theory, render the data on the drive irretrievable. Drive E takes this approach.
The advantage of this approach is that it is very fast: The sanitization command takes less than a second for Drive E. The danger, however, is that it relies on the controller to properly sanitize the internal storage location that holds the encryption key and any other derived values that might be useful in cryptanalysis. Given the bugs we found in some implementations of secure erase commands, it is unduly optimistic to assume that SSD vendors will properly sanitize the key store. Further, there is no way verify that erasure has occurred (e.g., by dismantling the drive).”

If not that serious - it belongs in the section about ‘Dark / offensive humour“

 

[pes]

But locally on my PC everything has to be stored on the same disc.

It is not. For example in linux creating a password for a user account creates a hash (with salt - which is a technique to make the hash stronger) a hash being the one way product of the function.
When you then log in your password is converted to the hash using the same algorithm and compared. If it is the same as the stored hash, you are logged in.
Calculating the password from the hash is not possible or requires years of brute force computing.

I wrote about this, referring to section 3.2.4 Encryption :

This is a different problem. I had to deal with this once and lost 500GB disk because the internal encryption key got corrupted. Amazing. The default encryption is transparent to the user, you do not even know it is there. It is automatically locked and unlocked. It basically does not offer much more than a faster way of destroying the data.

 

[UR20Z]

en using an external SSD or Flash drive there is no need to store the password on this disc. But locally on my PC everything has to be stored on the same disc. I think that the password to access my phone or tablet is not only in my brain, but also somewhere on the phone.

Only in its "hashed" form. (unless the device is hopelessly flawed, from a data-security viewpoint)

A hash algorithm is one to which you hand a piece of information - a password, an N-digit number, or whatever. The hash function accepts it, does some specialized math (amazingly simple math, to be honest) on it, and hands back another value that has no visible relationship to the input. The closer to "truly random" the output is, the better. The trick is this: The output of a hash algorithm that isn't flawed *CANNOT* be reversed to retrieve the information that was fed into the hash algorithm using methods and technology we have today. Perhaps in the future, this will change, but as of now, it can't be done. Currently, nobody knows (or at least, nobody admits to knowing) any method of doing that that can be accomplished in any amount of time significantly less than would be spent feeding every possible combination into the hash function and seeing if the value that comes out of it is the same as the hash that comes out when the original information is sent in. Further, hash functions are intentionally designed to make small changes in the input create HUGE changes in the output. So putting "Password1" into the function might spit out "F&W-(kLqaN6", but putting in "password1" (a single bit worth of difference to change the initial uppercase P into a lowercase p) might spit out "6^f]#elQmm", while putting in "Password2" (another 1-bit change from the original "Password1") might get you "n1X+Q%_5" - something that's nothing like either "6^f]#elQmm" or "F&W-(kLqaN6", aside from the fact that all three look like random gibberish.

Most modern (newer than *ROUGHLY* 2010) devices use this method to store passwords. When you tell them you want a new password, they request one from you, usually ask you to put it in again to make sure you didn't fat-finger it the first time, then hash it, destroy every trace of whatever you put in, and write the new hash over top of the old hash that resulted from your previous (if there was one) password.

So now you've got "F&W-(kLqaN6" (which is the hash of "Password1", but nobody except you knows that, and there is no currently known practical way to find out) stored on the phone, and you want to log on. You do whatever it takes to make your phone "wake up", and eventually, it asks you for your password. You key in "Password1" at the prompt, and the phone runs it through the hash function, then compares the result to what it has stored. Are they the same? If so, the phone unlocks, and you go about your biz. No match? Back to the "enter your password: " prompt.

I wrote about this, referring to section 3.2.4 Encryption :
“Many recently-introduced SSDs encrypt data by default, because it provides increased security. It also provides a quick means to sanitize the device, since deleting the encryption key will, in theory, render the data on the drive irretrievable. Drive E takes this approach.
The advantage of this approach is that it is very fast: The sanitization command takes less than a second for Drive E. The danger, however, is that it relies on the controller to properly sanitize the internal storage location that holds the encryption key and any other derived values that might be useful in cryptanalysis. Given the bugs we found in some implementations of secure erase commands, it is unduly optimistic to assume that SSD vendors will properly sanitize the key store. Further, there is no way verify that erasure has occurred (e.g., by dismantling the drive).”

The *ONLY* truly reliable way to destroy data that has been written to a hard drive, whether it's a spinning platter drive, or SSD, is physical destruction of the storage media - typically by running the entire unit through a grinder of some sort to reduce it to a pile of itty-bitty pieces that nobody, not even all the king's horses and all the king's men, can put back together in a functional form. The *REALLY* paranoid, like governments, will go even further - They'll melt the resulting pieces into slag, or stir them into a batch of concrete, or similar "Nope, not even gonna let you see the pieces" procedure.

 

[Kristy]

The wear distribution algorithm is not protecting data, it is protecting the drive. SSD and in general flash drives have a limited number of writes. Then the flash cells get damaged. So these algorithms spread the write instructions across the whole cell array to distribute writes and not overuse one portion of the disk. This extends the life of the drive. But it leaves copies of previously written data in those places since the algorithm does not in general keep track of the older copies, only the last one.

Do not even get me started on the """security""" of mobile devices.

No one should store anything on their phone they wouldn't want the whole world to see. A phone shouldn't be used to access this site either. Many people do, and without bothering to use a vpn, I'm sure.

So, to be clear, am I damaging my SSD drive if I'm using my digital shredder approx once a week to shred just one file?

 

[candychameleon]

There is smoke coming out of my ears after reading just a bit of this. My brain is fried.. Overworked my 3 remaining brain cells.

 

[UR20Z]

No one should store anything on their phone they wouldn't want the whole world to see. A phone shouldn't be used to access this site either. Many people do, and without bothering to use a vpn, I'm sure.

So, to be clear, am I damaging my SSD drive if I'm using my digital shredder approx once a week to shred just one file?

Yes.
But...

This is ALSO true for *EVERY* operation that writes to the SSD. Storing a bit - any bit, for any reason, by any mechanism the chip knows how to perform - inflicts a "quantum" of damage to the cell that bit gets stored into. How many quantums (quanta?) does it take to make that bit fail? I can't give you a number. I bet the uber-nerds can, though! (For the record: Hooray for uber-nerds - With the same sentiment as Hooray for rooftop koreans) But whatever the required quanta, eventually, it WILL fail if it's written to enough times. As noted previously, I'd suggest that one should consult an SSD uber-nerd for exactly what the value of "enough" is likely to be - I'm just a semi-quasi-nerd on that particular topic

 

[TheZipper]

No one should store anything on their phone they wouldn't want the whole world to see. A phone shouldn't be used to access this site either. Many people do, and without bothering to use a vpn, I'm sure.

So, to be clear, am I damaging my SSD drive if I'm using my digital shredder approx once a week to shred just one file?

Just curious, what’s wrong with using your phone to access this site (if you don’t download any files (I’m not even sure if you can?)) For example, all I do is look at videos and read posts. As far as I know, which isn’t much on this subject, the only thing anyone would be able to tell is that this phone accessed this site. Am I wrong? Thanks!

 

[dartel]

Just curious, what’s wrong with using your phone to access this site (if you don’t download any files (I’m not even sure if you can?)) For example, all I do is look at videos and read posts. As far as I know, which isn’t much on this subject, the only thing anyone would be able to tell is that this phone accessed this site. Am I wrong? Thanks!

Have you heard of a cache and temporary internet files?

 

[humananimal]

Have you heard of a cache and temporary internet files?

Temporary internet files is deleted at once you close a private browser. But as I read and wrote about, SSD and other digital discs, you can’t know if those files really is removed, or is just marked as free, and if the phone is stolen, criminals / authorities possibly can extract versions of the files.

So, to be clear, am I damaging my SSD drive if I'm using my digital shredder approx once a week to shred just one file?

I can’t answer for sure, but I am sure that this digital shredder will remove the normal access to the files, but as written above, you probably have several versions of the files at low level.

 

[dogluver101]

Temporary internet files is deleted at once you close a private browser. But as I read and wrote about, SSD and other digital discs, you can’t know if those files really is removed, or is just marked as free, and if the phone is stolen, criminals / authorities possibly can extract versions of the files.

I can’t answer for sure, but I am sure that this digital shredder will remove the normal access to the files, but as written above, you probably have several versions of the files at low level.

From using a FTK Imager to look at my M.2 drive after doing a full system reset, some files can still be recovered... Mainly what I found was old cookie files. I haven't used another software that also checks for file recovery though.

So, to be clear, am I damaging my SSD drive if I'm using my digital shredder approx once a week to shred just one file?

There is something like HW Monitor that can view the health of the SSD. Or may be any other out there.
I used CCleaner to delete free space up to 2 times, and my drive health is still in the top 95%.


If you want better protection, just use something like VeraCrypt. I personally use a crazy encryption method, and not the standard AES. I try to brute force my own encryption by using some software I found, including one that's by Elcomsoft. There was another that's run by CLI, but forgot the name and possibly lost the file.

 

[biburghboy]

Nothing some buckshot can't solve. Maybe with some tannerite.