Lossless compression for dummies

Maybe now some people can understand that Data Can Be Compressed And Still Not Lose Quality.

very nice! also microsoft is working hard on making a new compression technique which should be out next year :D

the only reason devs dont compress on blu ray is because they cant due to the read speedAwinagainov

Erm, better compression = lower read-speed requirements, but higher processor requirement. So Blu-Ray and the Cell are pretty much made for compression.

[QUOTE="Awinagainov"]the only reason devs dont compress on blu ray is because they cant due to the read speedGrive

Erm, better compression = lower read-speed requirements, but higher processor requirement. So Blu-Ray and the Cell are pretty much made for compression.

Thats not completely true. For compression to work, you need to be able to buffer enough data to uncompress it. If you read speed is slow, you won't be able to buffer. With uncompressed audio, you can just stream the data without the need for a buffer.

FLAC, WMA Lossless, and Dolby TrueHD are examples of lossless compression that is already out there. I think DTS does the same thing since it uses bitstreaming to expand surround effects to 8 channels. WAV is still the only true non-compressed lossless audio format, which uses pulse-code modulation at a sample rate of 48 kHz, or about 1.407 MB/s.Spindoc_SEI

TrueHD and DTS-Master Audio are the only 100% lossless audio formats for multichannel surround sound. (5.1+)

FLAC and WMA Lossless are 2 channel lossless audio formats. (2.0)

FLAC and WMA Lossless are 2 channel lossless audio formats. (2.0)m3Boarder32

Do you just make this stuff up as you go along? Totally wrong.

[QUOTE="Spindoc_SEI"]FLAC, WMA Lossless, and Dolby TrueHD are examples of lossless compression that is already out there. I think DTS does the same thing since it uses bitstreaming to expand surround effects to 8 channels. WAV is still the only true non-compressed lossless audio format, which uses pulse-code modulation at a sample rate of 48 kHz, or about 1.407 MB/s.m3Boarder32

TrueHD and DTS-Master Audio are the only 100% lossless audio formats for multichannel surround sound. (5.1+)

FLAC and WMA Lossless are 2 channel lossless audio formats. (2.0)

FLAC and WMA Lossless both support multichannel. You love to make up stuff, dont you?

of course! that solves all the problems! we can just compress and compress and compress ad infinitum, with no worry about ever hitting a ceiling no matter what we want in the game!

wait...

what happens when you need to compress more than that? new types of compression is needed if dvd9 is going to keep 360 going MS need to get evolving compression but at somepoint you simply can't compress anymore and it get too costly to find new ways to compress with out loss. that why BD is popular there is no need to compress anything more than needed.Ragashahs

Generally, 360's compressions are very efficient. What is described in this thread is the simplest kind of compression. More advanced compression algorithms compress by pattern, while others do it based on multiple passes of the compression.

what happens when you need to compress more than that? new types of compression is needed if dvd9 is going to keep 360 going MS need to get evolving compression but at somepoint you simply can't compress anymore and it get too costly to find new ways to compress with out loss. that why BD is popular there is no need to compress anything more than needed.Ragashahs

Please, please, please tell me where on the PS3 are you going to fit all of this magical uncompressed data in the 512 MB of memory that are present in both the 360 and the PS3.

[QUOTE="m3Boarder32"]FLAC and WMA Lossless are 2 channel lossless audio formats. (2.0)Velocitas8

Do you just make this stuff up as you go along? Totally wrong.

lmao :lol:

[QUOTE="Ragashahs"]what happens when you need to compress more than that? new types of compression is needed if dvd9 is going to keep 360 going MS need to get evolving compression but at somepoint you simply can't compress anymore and it get too costly to find new ways to compress with out loss. that why BD is popular there is no need to compress anything more than needed.XaosII

Please, please, please tell me where on the PS3 are you going to fit all of this magical uncompressed data in the 512 MB of memory that are present in both the 360 and the PS3.

it gets onloaded and offloaded as seamlessly as possible.

[QUOTE="XaosII"]

[QUOTE="Ragashahs"]what happens when you need to compress more than that? new types of compression is needed if dvd9 is going to keep 360 going MS need to get evolving compression but at somepoint you simply can't compress anymore and it get too costly to find new ways to compress with out loss. that why BD is popular there is no need to compress anything more than needed.Mordred19

Please, please, please tell me where on the PS3 are you going to fit all of this magical uncompressed data in the 512 MB of memory that are present in both the 360 and the PS3.

it gets onloaded and offloaded as seamlessly as possible.

With roughly comparable disc speeds, it doesnt make much of a difference between the two platforms. Its a big contributing reason why PS3/360 multiplats still have comparable loading times. The potential streaming capabilities of both systems in regards to games are identical.

And no developer will actually use uncompressed data or even purely lossless - low-rez textures off into the distance is already a lossy form of memory saving. There isnt enough memory to have the luxury to do that.It makes BD only useful for streaming, non-dynimic data with a very short lifespan, like the frames of a movie or a piece of dialogue.

Thats not completely true. For compression to work, you need to be able to buffer enough data to uncompress it. If you read speed is slow, you won't be able to buffer. With uncompressed audio, you can just stream the data without the need for a buffer.

rimnet00

Well, for most data used in games you don't necessarily need to stream: you can download the whole thing and store it in the system. Stuff like sound and video should be well withing the capabilities of any disc drive (otherwise movies would be unplayable).

I will start by confessing I have never heard of "lossless compression" and have no idea if that is what devs are actually using or not.

However, what I find ironic is that devs complain about the PS3 hardware because they say it is too difficult breaking their game into pieces to be calculated and the re-assembled asrequired...but this seems exactly what the compression does. So how is compressing data, by taking it apart and then reassembling as required, different from PS3 design? Why are developers having so much problem with PS3 but no problems with taking apart a game to compress.

Honest question.

[QUOTE="rimnet00"]

Thats not completely true. For compression to work, you need to be able to buffer enough data to uncompress it. If you read speed is slow, you won't be able to buffer. With uncompressed audio, you can just stream the data without the need for a buffer.

Grive

Well, for most data used in games you don't necessarily need to stream: you can download the whole thing and store it in the system. Stuff like sound and video should be well withing the capabilities of any disc drive (otherwise movies would be unplayable).

It's not nessesary, of course, but it is. When is the last time you 'installed' a game on a 360 or a PS3? Textures, sound clips, etc are in fact - believe it or not - streamed during gameplay. This is precisely why developers have been placing redudant data on BR disks, so as to see a statistical performance increase during stream times. By fragmenting and duplicating data onto the bluray (since space is plentiful) by probabal usage, you are allowing the BR head to constantly read, instead of jump around the disk. This increases the data flow of the bluray drive bus by a significant amount.

As for sound and video. It is compressed, even on HD-DVD and Bluray movies. The standard for video compression on these two formats is called H.264. As for the audio, there are a slew of compression codecs used, most famously the Dolby variety. There may be a few with completely uncompressed lossless audio, but I have yet to see one.

I will start by confessing I have never heard of "lossless compression" and have no idea if that is what devs are actually using or not.

However, what I find ironic is that devs complain about the PS3 hardware because they say it is too difficult breaking their game into pieces to be calculated and the re-assembled asrequired...but this seems exactly what the compression does. So how is compressing data, by taking it apart and then reassembling as required, different from PS3 design? Why are developers having so much problem with PS3 but no problems with taking apart a game to compress.

Honest question.

ZIMdoom

I gave reason for that on the first page. The PS3's slow disk read speed is not compression friendly, because compression requires buffering, and buffering requires faster read speeds. Compression is benefitial to storage space, not read speed; instead it's the opposite.

so why don't they still use CDs for games?Zero5000X

It's all statistics. That model would require way too much CPU speed to compensate for the uber compression required. It's all about balance between: compression, read speed, space, cpu power, etc

Maybe now some people can understand that Data Can Be Compressed And Still Not Lose Quality.

Wartzay

Eh, compression in its simpliest forms for dummies...but really its alot more complicated when i was learning it in ****.

[QUOTE="ZIMdoom"]

I will start by confessing I have never heard of "lossless compression" and have no idea if that is what devs are actually using or not.

However, what I find ironic is that devs complain about the PS3 hardware because they say it is too difficult breaking their game into pieces to be calculated and the re-assembled asrequired...but this seems exactly what the compression does. So how is compressing data, by taking it apart and then reassembling as required, different from PS3 design? Why are developers having so much problem with PS3 but no problems with taking apart a game to compress.

Honest question.

rimnet00

I gave reason for that on the first page. The PS3's slow disk read speed is not compression friendly, because compression requires buffering, and buffering requires faster read speeds. Compression is benefitial to storage space, not read speed; instead it's the opposite.

That's not what I asked though. It appears to me that lossless compression involves writing a special code, disassembling part of your data (in this case your game data), then re-assembling your data when you need it.

The PS3 cell is meant to work best when you code the game as seperate parts and the re-assemble them when you need them. The data isn't compressed at all, just broken apart overseperate processors, then stuck back together at the end.

Developers complain that this process is too difficult for the PS3...but it looks like the compression works pretty much the same way and yet developers may be fine with that.

I guess the questions are (1) Is lossless compression actually being used now in gaming, (2) can it save enough space when we are talking about gigs of data, (3) who does the compression? the devs or do they use a program or get someone else to do it? (4) Why bother compressing at all, unless absolutely necessary, considering it is more work than not compressing since you have to write special code for the compression.

[QUOTE="rimnet00"][QUOTE="ZIMdoom"]

I will start by confessing I have never heard of "lossless compression" and have no idea if that is what devs are actually using or not.

However, what I find ironic is that devs complain about the PS3 hardware because they say it is too difficult breaking their game into pieces to be calculated and the re-assembled asrequired...but this seems exactly what the compression does. So how is compressing data, by taking it apart and then reassembling as required, different from PS3 design? Why are developers having so much problem with PS3 but no problems with taking apart a game to compress.

Honest question.

ZIMdoom

I gave reason for that on the first page. The PS3's slow disk read speed is not compression friendly, because compression requires buffering, and buffering requires faster read speeds. Compression is benefitial to storage space, not read speed; instead it's the opposite.

That's not what I asked though. It appears to me that lossless compression involves writing a special code, disassembling part of your data (in this case your game data), then re-assembling your data when you need it.

The PS3 cell is meant to work best when you code the game as seperate parts and the re-assemble them when you need them. The data isn't compressed at all, just broken apart overseperate processors, then stuck back together at the end.

Developers complain that this process is too difficult for the PS3...but it looks like the compression works pretty much the same way and yet developers may be fine with that.

I guess the questions are (1) Is lossless compression actually being used now in gaming, (2) can it save enough space when we are talking about gigs of data, (3) who does the compression? the devs or do they use a program or get someone else to do it? (4) Why bother compressing at all, unless absolutely necessary, considering it is more work than not compressing since you have to write special code for the compression.

The cell is good at handling multiple independent processes not seperate-dependent pieces of data.

As for the complexity of compression, it's something that can be a) providing by Microsoft/Sony b)purchased from a third party c) written by one team and used amongst the development team / the entire organization very easily. While on the other hand, statically allocating data across sectors of a disk is a lot harder, since it is dependent on the final product.

As for your specific questiosn:
1. Yes it's being used, and Microsoft uses it more then Sony
2. You can save a ton of space, hense the reason why Microsoft decided to stick with DVD9s instead of the new media types (bluray / hddvd)
3. I answered in the paragraph before. Microsoft internal dev, third parties, the dev teams themselves.
4. It depends on the situation. Sometimes its better to use more space and save CPU cycles. Sometimes its better to use more cpu cycles and less space. It's all statistical. However, in the scenario between the two consoles, I think it is obvious which was the better way to go.

[QUOTE="Zero5000X"]so why don't they still use CDs for games?rimnet00

It's all statistics. That model would require way too much CPU speed to compensate for the uber compression required. It's all about balance between: compression, read speed, space, cpu power, etc

Which starts to get us to the heart of the matter. The hermits actually have it best. This is because loading everything onto the HDD is faster/more efficient than reading off any portable disk media be it CD, DVD9 or BD. This model works well for the PC because they typically have large HDDs and plenty (relatively) space to store games.

A hybrid model in between would be to read off the portable disk but preload some onto the HDD. This is in fact what the PS3 does in many cases to improve speed.

Now we get to the business case.

All PS3s have an HDD, and they are all upgradeable for cheap by the user

All PS3s also have a BD

Delays in the manufacturing/availability of BD diodes delayed the PS3 at least 6 months

The BD is the single most expensive component of the PS3 and added a minimum $150 cost....likely considerably more

HDD size doubles about every 6 months to a year....because of this HDDs are quite cheap. Indeed, HDD storage space has always been relatively cheap in the past 15 years and not a huge barrier to the pace of computer development.

You and I can buy a 250 GB HDD for $175...and falling

Sony, MS or Ninetendo could buy 250GB HDDs en massefor a heck of a lot less than you or I

Sony's cost for any of their versions of their HDD + the cost of the BD drive is easily more than Sony's hypothetical cost for equipping 250GB HDDs + a DVD9 drive

Therefore, Sony could have:

Launched with a huge HDD,

at a lower price than what they did,

and 6 months earlier,

and still 'solved' the storage space issues (real or imagined)

with an overall performance better than either DVD9 (without HDD) or BD (with or without HDD)

using a hybrid DVD9 + HDD strategy (similar to their BD + HDD strategy)

and, they still could have followed a MS strategy and made the BD player an add-on

BUT THEY DIDN'T....AND IT ISN'T BECAUSE OF A GAMING REASON, NOR A PS3 FINANCIAL REASON

That's not what I asked though. It appears to me that lossless compression involves writing a special code, disassembling part of your data (in this case your game data), then re-assembling your data when you need it.ZIMdoom

That's one type of lossless compression. Another type (that you see in places like zip files) involves finding "good" pattern of bits and using a marker to represent that pattern: say, if you have a lot of 000000's in your code, you can represent it as a "2" and save 4 bits every time you encounter the sequence.

The PS3 cell is meant to work best when you code the game as seperate parts and the re-assemble them when you need them. The data isn't compressed at all, just broken apart overseperate processors, then stuck back together at the end.ZIMdoom

Plus the processing code itself is usually not compressed; code compilation already makes it pretty efficient in terms of space and you're not going to get much more out of it, not with the CPU/memory overhead required. Media is always the first target of compression.

Developers complain that this process is too difficult for the PS3...but it looks like the compression works pretty much the same way and yet developers may be fine with that.

I guess the questions are (1) Is lossless compression actually being used now in gaming, (2) can it save enough space when we are talking about gigs of data, (3) who does the compression? the devs or do they use a program or get someone else to do it? (4) Why bother compressing at all, unless absolutely necessary, considering it is more work than not compressing since you have to write special code for the compression.ZIMdoom

I highly doubt developers invent their own compression schemes when entire research branches of computer science are devoted to the topic. That said, lossless compression does not save enough space for a 25GB to turn into a 9GB one without loss; something like WMA lossless will turn your 50MB .wav file into a 25MB .wma file, but pales in comparison to a 5MB .mp3 file in terms of disk savings.

People make such a huge deal over lossless encoding sometimes; honestly, the reason why media compression is lossy is because people can tolerate the difference. Are people complaining about 700MB DiVX rips, 320kps MP3 rips, or 90 quality JPEG's? Then why care?

I will start by confessing I have never heard of "lossless compression" and have no idea if that is what devs are actually using or not.

However, what I find ironic is that devs complain about the PS3 hardware because they say it is too difficult breaking their game into pieces to be calculated and the re-assembled asrequired...but this seems exactly what the compression does. So how is compressing data, by taking it apart and then reassembling as required, different from PS3 design? Why are developers having so much problem with PS3 but no problems with taking apart a game to compress.

Honest question.

ZIMdoom

Actually its easy to break fairly independent processes into different threads. Most 360/PS3 games use a different thread for renderer, audio, physics, AI, decompression etc. The hard part is breaking something like rendering into multiple threads. Since the PS3 has 8 different cores, each fairly weak in themselves, you absolutely have to multithread rendering since it takes up the vast majority of processing power.

[QUOTE="rimnet00"]

[QUOTE="Zero5000X"]so why don't they still use CDs for games?SUD123456

It's all statistics. That model would require way too much CPU speed to compensate for the uber compression required. It's all about balance between: compression, read speed, space, cpu power, etc

Which starts to get us to the heart of the matter. The hermits actually have it best. This is because loading everything onto the HDD is faster/more efficient than reading off any portable disk media be it CD, DVD9 or BD. This model works well for the PC because they typically have large HDDs and plenty (relatively) space to store games.

A hybrid model in between would be to read off the portable disk but preload some onto the HDD. This is in fact what the PS3 does in many cases to improve speed.

Now we get to the business case.

All PS3s have an HDD, and they are all upgradeable for cheap by the user

All PS3s also have a BD

Delays in the manufacturing/availability of BD diodes delayed the PS3 at least 6 months

The BD is the single most expensive component of the PS3 and added a minimum $150 cost....likely considerably more

HDD size doubles about every 6 months to a year....because of this HDDs are quite cheap. Indeed, HDD storage space has always been relatively cheap in the past 15 years and not a huge barrier to the pace of computer development.

You and I can buy a 250 GB HDD for $175...and falling

Sony, MS or Ninetendo could buy 250GB HDDs en massefor a heck of a lot less than you or I

Sony's cost for any of their versions of their HDD + the cost of the BD drive is easily more than Sony's hypothetical cost for equipping 250GB HDDs + a DVD9 drive

Therefore, Sony could have:

Launched with a huge HDD,

at a lower price than what they did,

and 6 months earlier,

and still 'solved' the storage space issues (real or imagined)

with an overall performance better than either DVD9 (without HDD) or BD (with or without HDD)

using a hybrid DVD9 + HDD strategy (similar to their BD + HDD strategy)

and, they still could have followed a MS strategy and made the BD player an add-on

BUT THEY DIDN'T....AND IT ISN'T BECAUSE OF A GAMING REASON, NOR A PS3 FINANCIAL REASON

"sigh", what could have been...

[QUOTE="m3Boarder32"]

[QUOTE="Spindoc_SEI"]FLAC, WMA Lossless, and Dolby TrueHD are examples of lossless compression that is already out there. I think DTS does the same thing since it uses bitstreaming to expand surround effects to 8 channels. WAV is still the only true non-compressed lossless audio format, which uses pulse-code modulation at a sample rate of 48 kHz, or about 1.407 MB/s.rimnet00

TrueHD and DTS-Master Audio are the only 100% lossless audio formats for multichannel surround sound. (5.1+)

FLAC and WMA Lossless are 2 channel lossless audio formats. (2.0)

FLAC and WMA Lossless both support multichannel. You love to make up stuff, dont you?

I have seen him done this tons of time

What I don't understand is how are PS3 games supposed to look better if the video memory is only 256MB.

Even by streaming.

And if you push the system too hard, you run into framerate and texture popping issues.

PC games don't have lossless audio...PC Games don't use FLAC or WMA Lossless, they use DTS Connect and Dolby Digital Live. Both of which are not lossless.m3Boarder32

aren't you the same guy that said FLAC and WMA lossless only do 2 channel?

PC games don't have lossless audio...PC Games don't use FLAC or WMA Lossless, they use DTS Connect and Dolby Digital Live. Both of which are not lossless.m3Boarder32

the only reason devs dont compress on blu ray is because they cant due to the read speedAwinagainov

[QUOTE="m3Boarder32"]PC games don't have lossless audio...PC Games don't use FLAC or WMA Lossless, they use DTS Connect and Dolby Digital Live. Both of which are not lossless.Ontain

aren't you the same guy that said FLAC and WMA lossless only do 2 channel?

I was half right, FLAC is virtually impossible to get 5.1 out of. WMA Lossless is easier.

Besides what does that have to do with PC Games not having lossless audio? Keep braging about being able to download and play FLAC and WMA Lossless files :shock:

What I don't understand is how are PS3 games supposed to look better if the video memory is only 256MB.

Even by streaming.

And if you push the system too hard, you run into framerate and texture popping issues.

BlueBarad

[QUOTE="BlueBarad"]

What I don't understand is how are PS3 games supposed to look better if the video memory is only 256MB.

Even by streaming.

And if you push the system too hard, you run into framerate and texture popping issues.

AIH_PSP

Wow

I will start by confessing I have never heard of "lossless compression" and have no idea if that is what devs are actually using or not.

However, what I find ironic is that devs complain about the PS3 hardware because they say it is too difficult breaking their game into pieces to be calculated and the re-assembled asrequired...but this seems exactly what the compression does. So how is compressing data, by taking it apart and then reassembling as required, different from PS3 design? Why are developers having so much problem with PS3 but no problems with taking apart a game to compress.

Honest question.

ZIMdoom

Sure. Breaking code into independent processes really isn't that hard. What's very hard is called parallelism - finding tasks that can run at the same time. In video games, this is very difficult - the graphics and sound depend on where everything is in the world, which is affected by the AI, which is affected by the physics, which are affected by the user's input as well as incoming network data.

A second concern with highly parallel code is locking - making sure that no two pieces of code modify the same data at the same time, and that code wanting to read that data waits for it to be updated. Avoiding contention between threads is a critical part of multithreaded programming.

As for compression, it relies on substitution within a single piece of data - imagine the phrase "How now, brown cow." Replace "ow" with "x" and you get "Hx nx, brxn cx.", which is 8 characters shorter. You still need a lookup table, however, to restore the data. So it really looks like "x:ow|Hx nx, brxn cx." Here's where it gets ugly - that's actually longer than the original! Compression relies on a lot of repeated data to really become efficient. (There are also far more advanced forms of compression than the one I just described, but the basic principle is the same.)

So how is decompression different? Why can you break the data up into chunks and not have contention issues? Because everything you need to decompress the data is right there in the data. You don't have to rely on anything else - just crack open enough space to hold the data, and blast it in.

Do you have ANY idea how big lossless compressed media is, even with the latest codecs?

Uncompressed audio and video take hundreds of gigabytes, even with blu-ray that's impossible(not to mention stupid).

Just face the facts, Blu-ray space allows better quality media, that's something that can't be spinned.

And quality of the media has ABSOLUTELY nothing to do and absolutely doesn't interfere in the quality of the games per se. But having a great game with great audio and video definetly is better, if you have high quality hardware to take advantage.

Do you have ANY idea how big lossless compressed media is, even with the latest codecs?

Uncompressed audio and video take hundreds of gigabytes, even with blu-ray that's impossible(not to mention stupid).

Just face the facts, Blu-ray space allows better quality media, that's something that can't be spinned.

And quality of the media has ABSOLUTELY nothing to do and absolutely doesn't interfere in the quality of the games per se. But having a great game with great audio and video definetly is better, if you have high quality hardware to take advantage.

lordxymor

Heavenly Sword has over 10GB of Audio. The audio alone wouldn't fit on a DVD.

seeing some people recently have tried to bring up the argument of DVD9 / 360 compression causing a decrease in visual fidelity I thought I'd bring this thread back up.

both the PS3 and the 360 make use of LOSSLESS compression. it means exactly what you'd think.

any visual oddities / degradations / poor quality you notice is due to the limitations of the console the game is being run on.

This needed to be said... hopefully this'll help stop the cries of "buh buh buh teh compression!"Funkyhamster

This is system wars. When did having solid proof of anything matter? People just ignore it.

OP, this was not obvious? It's like making a ZIP file. Dreams-Visions

this thread is a year old - i dug it up because Dan_K was trying to argue that DVD9 and compression was holding back PS3 multiplats and causing graphical degradation. was in the '360 holding back PS3 potential' thread or something like that ;) just want people to know that lossy compression isn't used this gen.

you are correct though in your analogy, lossless compression is analogous to zip / rar compression. but this unfortunately isn't obvious for all users in SW. some people think it's a problem, and that is bleeds in to PS3 (which makes so little sense on so many levels seeing the data is stored differently on BR anyways).

the only issues with lossless are access time, i.e. loading time. which i should point out the PS3 has the most issues with. hence the installs to HDD.