Agartha Dreamcast reverse engineering
To read on a PC:
Now most of the work I've done so far on it is just preliminary. To start off, I'm focusing on the April 18th release, which may be found here . The first obstacle was just getting the image mounted on my Linux machine. CDI images are proprietary, so usually I just convert them into an iso file and mount will be happy, but these images would simply not convert using any software that I knew. I finally managed to get the contents open using gditools , which I think is written by members of this forum? if so - thanks!
So on the filesystem root we have a bunch of files:
256344068 Apr 17 2001 000DUMMY.DAT
256344068 Apr 17 2001 0AGARTHA_DEMO2001.MPG
1105126 Apr 17 2001 1AGARTHA.BIN
3458 Mar 9 2001 ADXLISTDEMO.LST
36019 Apr 17 2001 AGARTHA.LST
13144526 Apr 17 2001 AGARTHA.PAK
18608 May 18 2000 AUDIO64.DRV
1021590 Mar 8 2001 BODNATH_EXTR01.ADX
1364634 Mar 8 2001 BODNATH_EXTR02.ADX
1311174 Feb 28 2001 BODNATH_EXTR08.ADX
1616130 Feb 28 2001 BODNATH_EXTR09.ADX
2110518 Feb 28 2001 GOKYO_EXTR01.ADX
1170810 Feb 28 2001 GOKYO_EXTR07.ADX
3225258 Feb 28 2001 GOSAINKUND_MOD_NIV_SHORT.ADX
1943910 Feb 28 2001 KALAPATHAR_EXTR01.ADX
1553346 Feb 28 2001 KALAPATHAR_EXTR03.ADX
992394 Feb 28 2001 KALIGANDAKI_EXTR03.ADX
1037214 Feb 28 2001 KALIGANDAKI_EXTR07.ADX
1254042 Feb 28 2001 KALIGANDAKI_EXTR09.ADX
1122066 Feb 28 2001 KALIGANDAKI_EXTR11.ADX
5826294 Feb 28 2001 KANCHENJUNGA_EQ.ADX
1533150 Feb 28 2001 KANCHENJUNGA_EXTR09.ADX
1560114 Feb 28 2001 KANCHENJUNGA_EXTR11.ADX
1130742 Feb 28 2001 KANCHENJUNGA_EXTR13.ADX
2034630 Feb 28 2001 KIANGJING_EXTR03.ADX
219474 Feb 28 2001 KIANGJING_EXTR06.ADX
1721718 Feb 28 2001 KIANGJING_EXTR07.ADX
860022 Feb 28 2001 LADAKH_EXTR02.ADX
2196882 Mar 9 2001 LOUPIOTV3.ADX
2159300 Feb 26 2001 MOULIN.F3D
1739538 Feb 28 2001 PE03_22_EXTR01.ADX
4096 May 15 18:24 RESS
Coded:
//Agartha.PAK template
LittleEndian ();
// one per entry in AGARTHA.LST
SetBackColor (cPurple);
const uint NUM_FILES = 550;
uint offsets [NUM_FILES];
// before each file
struct File
{
uint32 uncompressed;
uint32 compressed;
uint16 flags;
byte data [compressed];
};
local uint i = 0;
while (i <NUM_FILES)
{
FSeek (offsets [i]);
if (i% 2! = 0) SetBackColor (cLtPurple);
else SetBackColor (cDkPurple);
File file;
i ++;
}
The first interesting bit is the .mpg file, which is a 15 minute long video of the game, which is posted on @Sega Dreamcast Info 's site. The 000DUMMY.DAT is the same file. I guess it's there to pad out the disk. Weird, but I've seen weirder. The ADX files are all music tracks, best as I can tell. They are playable using VLC or ffplay, but some seem broken or unsupported. The RESS directory is curiously sparse, so this means the assets are hiding somewhere ... but where?
AGARTHA.PAK, of course.
However, when I popped it open in my hex editor it was immediately apparent this would be a more difficult task. The PAK file structure has no file manifest or anything, it's essentially just a big blob of data, yet there must be some way of mapping files to offsets somewhere. That's when I looked at AGARTHA.LST.
AGARTHA.LST is a big text file, with a bunch of file names inside it, one per line. This got me thinking, so I went back to my hex editor and sure enough, at the top of the file was a big blob of uints, and more interestingly, their values were all monotonic. I started jumping around inside the PAK file, treating each uint as an offset; sure enough, at each location was what appeared to be different files, and in the case of the PVR textures, their headers all lined up with the file names. This was the first important discovery in the journey of unlocking this treasure chest: The AGARTHA.LST contains a list of all files inside the AGARTHA.PAK; each file is written inside the PAK in the order it appears in the LST. There are 550 files total.
How ever, there's another big snag: the files are all compressed in what appears to be some form of LZW, or more probably, LZSS encoding. The give away here is that the early parts of the file are quite legible, but by the end of each file it's a binary mess. So in order to continue with the task of reversing Agartha, I have to figure out the decompression scheme. fortunately, LZW / LZSS isn't exactly black magic, but there are a ton of variations and at the moment I'm unsure of how to continue, so I have to ask: does anyone got any tips on reversing compression algorithms?
I gave it a try, and about half of the ADX files still fail to convert. I popped them open in the hex editor and it looks like the ones that fail are completely different formats all together. I looked up the ADX format to be sure, and they definitely do not match any known version of it. I did find something that might be interesting, though, see attached image. The valid ADX files do not have this section, so I wonder what's going on here.
I'll try a hand at cracking the decompression this weekend. It's going to feel great once we can get this treasure chest opened up.
Alright, I sat down and wrote out a script to dump the PAK file. So far it recreates the file structure and dumps the raw binaries to disk, but the compression remains a work in the progress. I did however learn that files are stored in at least 3 different modes, one of which is uncompressed. It's unclear if this specifies a compression quality setting or different algorithm at the moment.
Here's the script for anyone interested in examining the raw data:
Python:
#! / usr / bin / env python3
from struct import unpack, calcsize
import os
def uncompress (data):
return data
def main ():
PREFIX = 'AGARTHA'
DEST = 'content'
#Load the manifest
with open (PREFIX + '.LST' , 'rb' ) as f:
count = 0
lines = f.readlines ()
#Build a list of files
files = []
for ln in lines:
#Remove blanks and comments
ln = ln.decode ( 'latin' ) .strip () .lower () .replace ( '/' , '\\' )
yew '' == ln:
keep on going
yew '#' == ln [: 1 ]:
keep on going
#Insert the path into the list for later
files.append (ln)
count + = 1
#Figure out the common prefix
pref = os.path.commonprefix (files) .replace ( '\\' , '/' )
print ( f'Common Prefix: " {pref} " ' )
#Build a path list ...
paths = []
for p in files:
sanitised = os.path.normpath (p.replace ( '\\' , '/' ))
paths.append (sanitised.replace (pref, '' ))
#Extract files from archive ...
with open (PREFIX + '.PAK' , 'rb' ) as f:
offsets = unpack ( f '< {count} I' , f.read (calcsize ( f '< {count} I' )))
for i in range (count):
#Rebuild path on disk
fn = os.path.basename (paths [i])
dn = os.path.join (DEST, os.path.dirname (paths [i]))
os.makedirs (dn, exist_ok = True )
#Seek to find and read header
f.seek (offsets [i])
uncompressed, compressed, mode = unpack ( '<IIH' , f.read (calcsize ( '<IIH' )))
#Read and uncompress contents as needed
rate = 100 * compressed / uncompressed
MODES = [ 'uncompressed' , 'UNK0' , 'UNK1' ]
print ( f'Uncompressing " {paths [i]} ", ratio: {rate: .4} % ( {MODES [mode]} ) ' )
data = uncompress (f.read (compressed))
#Write it to disk and free data
with open (os.path.join (dn, fn), 'wb' ) as out:
out.write (data)
del data
if __name__ == "__main__" :
main ()
Addendum:
The encoding is definitely LZSS, at least the 0x02 mode. By complete serendipity I found out that it follows a similar scheme to the algorithm used in Allegro, where a control byte is issued for every 8 tokens, where a token may be a 1 byte literal or a 2 byte base / length pair. So at most there may be 16 bytes of data between control bytes. I haven't quite got it working though.
Addenum II:
I wrote out a routine to decompress the LZSS stuff as described above. It works for files encoded using Allegro, but only works partially for Agartha, so this seems to confirm to me that Agartha uses a variant. One observation I noticed is that I have to flip the byte order of the base / length pairs to get the proper number of bytes to decode, but I haven't quite managed to exactly get the proper bytes out of it. Single byte literals decompress perfectly, so there's no question about the over all scheme being similar to Allegro. It's just a matter of how to interpret the base index ...
ADXLISTDEMO.LST
To the best of my knowledge this is just a text file containing a list of music tracks, the same ones sitting on the disk root. One track per line, windows style line endings. I use this one to test since it's easy to tell when you get something right or wrong.
Unfortunately it seems to be more than just an endian issue, but it's too early to rule it out completely.
As an aside, I want to point out how perfect this algorithm is for the Dreamcast. For those unaware, the GD rom drive is quite slow and painful, and iirc, it's connected to the main cpu over the dainty G1 bus, so bandwidth is at a premium, but more than that, the DC's main cpu has a special feature that lets the programmer turn half the cache into super fast general purpose memory, which is perfect for this algorithm, which only needs a ring buffer and a few variables to keep track of its state. The Agartha team knew what was up!
Coded:
Ress\Music\Demo2001\bodnath_extr08.wav Ress\Music\Demo2001\bodnath_extr02.wav Ress\Music\Demo2001\kiangjing_extr08.wav Ress\Music\Demo2001\gokyo_extr04.wav Ress\Music\Demo2001\bodnath_extr11.wav Ress\Music\Demo2001\bodnath_extr12.wav Ress\Music\Demo2001\bodnath_extr10.wav Ress\Music\Demo2001\bodnath_extr05.wav Ress\Music\Demo2001\bodnath_extr06.wav Ress\Music\Demo2001\bodnath_extr04.wav Ress\Music\Demo2001\bodnath_extr13.wav Ress\Music\Demo2001\bodnath_extr14.wav Ress\Music\Demo2001\bodnath_extr09.wav Ress\Music\Demo2001\bodnath_extr07.wav Ress\Music\Demo2001\bodnath.wav Ress\Music\Demo2001\bodnath_extr01.wav Ress\Music\Demo2001\bodnath_extr03.wav Ress\Music\Demo2001\gokyo_extr07.wav Ress\Music\Demo2001\gokyo_extr06.wav Ress\Music\Demo2001\gokyo_extr05.wav Ress\Music\Demo2001\gokyo_extr08.wav Ress\Music\Demo2001\gokyo.wav Ress\Music\Demo2001\gokyo_extr01.wav Ress\Music\Demo2001\gokyo_extr02.wav Ress\Music\Demo2001\gokyo_extr03.wav Ress\Music\Demo2001\Gosainkund_mod_niv_short.wav Ress\Music\Demo2001\kalapathar.wav Ress\Music\Demo2001\kaligandaki.wav Ress\Music\Demo2001\kalapathar_extr05.wav Ress\Music\Demo2001\kanchenjunga_extr11.wav Ress\Music\Demo2001\kanchenjunga_extr15.wav Ress\Music\Demo2001\kanchenjunga_extr05.wav Ress\Music\Demo2001\kanchenjunga_extr09.wav Ress\Music\Demo2001\kanchenjunga_extr12.wav Ress\Music\Demo2001\kanchenjunga_extr16.wav Ress\Music\Demo2001\kanchenjunga_extr06.wav Ress\Music\Demo2001\kanchenjunga_extr13.wav Ress\Music\Demo2001\kanchenjunga_extr07.wav Ress\Music\Demo2001\kanchenjunga_eq.wav Ress\Music\Demo2001\kanchenjunga_extr10.wav Ress\Music\Demo2001\kanchenjunga_extr14.wav Ress\Music\Demo2001\kanchenjunga_extr08.wav Ress\Music\Demo2001\kaligandaki_extr12.wav Ress\Music\Demo2001\kaligandaki_extr13.wav Ress\Music\Demo2001\kaligandaki_extr10.wav Ress\Music\Demo2001\kaligandaki_extr11.wav Ress\Music\Demo2001\kaligandaki_extr06.wav Ress\Music\Demo2001\kaligandaki_extr07.wav Ress\Music\Demo2001\kaligandaki_extr05.wav Ress\Music\Demo2001\kaligandaki_extr14.wav Ress\Music\Demo2001\kaligandaki_extr15.wav Ress\Music\Demo2001\kaligandaki_extr08.wav Ress\Music\Demo2001\kaligandaki_extr09.wav Ress\Music\Demo2001\kalapathar_extr01.wav Ress\Music\Demo2001\kalapathar_extr02.wav Ress\Music\Demo2001\kalapathar_extr03.wav Ress\Music\Demo2001\kalapathar_extr04.wav Ress\Music\Demo2001\kaligandaki_extr01.wav Ress\Music\Demo2001\kaligandaki_extr02.wav Ress\Music\Demo2001\kaligandaki_extr03.wav Ress\Music\Demo2001\kaligandaki_extr04.wav Ress\Music\Demo2001\kanchenjunga_extr01.wav Ress\Music\Demo2001\kanchenjunga_extr02.wav Ress\Music\Demo2001\kanchenjunga_extr03.wav Ress\Music\Demo2001\kanchenjunga_extr04.wav Ress\Music\Demo2001\kiangjing_extr10.wav Ress\Music\Demo2001\kiangjing.wav Ress\Music\Demo2001\kiangjing_extr04.wav Ress\Music\Demo2001\kiangjing_extr11.wav Ress\Music\Demo2001\kiangjing_extr01.wav Ress\Music\Demo2001\kiangjing_extr02.wav Ress\Music\Demo2001\kiangjing_extr03.wav Ress\Music\Demo2001\kiangjing_extr07.wav Ress\Music\Demo2001\kiangjing_extr05.wav Ress\Music\Demo2001\kiangjing_extr06.wav Ress\Music\Demo2001\kiangjing_extr09.wav Ress\Music\Demo2001\ladakh.wav Ress\Music\Demo2001\ladakh_extr03.wav Ress\Music\Demo2001\ladakh_extr01.wav Ress\Music\Demo2001\ladakh_extr02.wav Ress\Music\Demo2001\Patshupatinath_modif_niv.wav Ress\Music\Demo2001\Pe03_22_extr01.wav Ress\Music\Demo2001\loupiotV3.wav
I figured it out.
Here's the trick:
the pairs are 16 bits wide, written little endian; the lower 4 bits are the length - 3, and the remaining 12 are the unsigned base. when you decode the pair, subtract base + 1 from the ring buffer position and cache it. this will be your offset, now just read out length bytes from the ring buffer relative to the offset.
I ran some metrics on the files, and about 60% are encoded in LZSS, 30% in some other encoding - probably LZW - and 10% are uncompressed. Here are a few bits that I've found so far. I've employed the PVR routines that I wrote for Castlevania again, so we can see the textures ...
A lot of Agartha appears to be scriptable in plain text, very similar to how Castlevania was. Again, this is probably because DeLoura's book was the hot read back then!
Coded:
/***************************************************************************** ______________________________________________________________________ ) ) / Agartha - Demo2001 / / Playable.def / (_____________________________________________________________________( | ) | | Purpose / Palyable demo. Player can walk through Agartha world | | ( as well as meet some characters... |==========\==========================================================| | ) | | Author / Sébastien Viannay - NoCliché | |=========(===========================================================| | \ | | History ) 01/03/2001 : Creation. | |__________/__________________________________________________________| *****************************************************************************
/ // include the entry point scene for the playabale demo part
INCLUDE "Village\Village.def"
Some sounds:
Kirk (the hero) texture:
Kirk's hair
Kirk's head
Kirk's jacket
kirk bag
Book (Lore)
Page 1
In the beginning, the massive earthquake, that the Bible erroneously calls Flood,
deeply transformed terrestrial geography and led the 13 tribes on the path of great migrations.
Guided by those that the Pnakotics Manuscrits (Vatican libraries archives) have designed under the generic name of "the 9 unknown ones", those tribes populated the highlands that escaped the rising waters.
That's how various cyclopean cities were born. Ancient papyrus preserved in the archeological museum of Cairo and bearing the seal of Meneptah (1200 years before our era) mention for example the forbidden city of the highlands of Leng which has seemingly entirely vanished following the cataclysm that shook the region of Pamir.
Page 2
It was during risky peregrinations, that led by Hassuna, the Sennacherib clan, also called children of the first world, discovered the gigantic schist grottoes that formed organically not far from the higher spring of the Brahmaputra (Himalayan Chain), at 6700 meters above the current sea level.
The Sennacherib decided to settle and take root there. One can find multiple allusion to this fact in the Mahabharata (Third Book)
That's probably how an underground passage was found, linked to a complex network of passageways leading to the earth core, legendary location in multiple mythologies that refer to what we know today as the Hollow Earth.
Page 3
Following the guidance of the "Enlightened One", the clan decided after the ritual sacrifices to build a secret city that could shelter a part of the human species should a cataclysmic event happen. No one knows exactly what techniques were used to accomplish this "Grand Design", or Agartha in Tibetan, which also signifies:
The subterranean kingdom at the center of the Earth where the King of the World reigns.
The building of the city was spread over numerous generations, and it seems, based on the few documents that have survived, that Agartha was never completely built. Local folklore (such as the chantings of Lapcha) attributes to the "inevitable devastation", this frenzy that led to enslavement and death a lot of the builders of this subterranean metropolis. Although it seems that events far more dramatic than those prophecies preceded this titanic work.
Page 4
Agartha was the home of powerful malevolent forces fallen and banished from the divine kingdom. Under the divine curse prohibiting them from appearing in the open, those infernal forces have been buried in the deepest part of the earth.
Planning their return on Earth, they started to decimate the children of the first world using an unknown disease. By doing this they made Agartha, an antechamber to hell in which they reign supreme.
Leading the demoniac hordes, is the one they call The Sentinel (Y'aga Heer'GHta in the Sennacherib dialect). If the writings are to be trusted, the city of Agartha communicates and opens towards other heavens (please refer to the works of Julius Horbiger, Bale faculty 1724 and especially the corpus 46 entitled "On the Hollow Earth")
Page 5
Indeed it seems that the children of the first world had planned to return to the surface after the “inevitable devastation” that they feared. Survivors of the epidemic were able to transmit to other people some of their knowledge. That's why Gozer's tablets and “Dead Sea Scrolls” allude to the existence of a “relay that leads soul from the surface to the depths below”.
Once a century, when the Scorpio enters the Virgo square, a chosen woman is born. A direct descendent of the Sennacherib family, she represents the ultimate salvation or the ultimate threat that can either save or doom the land. Indeed, with the chosen one's sacrifice, carried out under certain conditions, the Sentinel can either rise in broad daylight or be cast back to its lair.
Nowadays, the only custodians of this knowledge and undertakers of this task, have gathered in a secret society, called the “Order of the Sennacherib”. Without being descendents of the clans, their mission is to fight Evil. Their sworn enemies, hell bent on awakening the Sentinel, are called “the Conspirators of Twilight”
English translation by Vince
Page 1
Page 2
Page 3
Page 4
Page 5
Crawley book
The animation sciptes of the video
Each object can have a life (a script) (the term dates from alone in the dark). It also has tracks that tell the character to move from flag to flag. The "flag" option (the flags with the No Cliché logo) is in the debug menu, the same for "life" from before. For Agartha (or Toy Commander) they added notions of camera and splines to make camera movements.
example animation:
OBJECT hunter
{
Caps = LIFE | ANIM | MOVE SpeedRot = 16384
GenName = Hunter
Track = Food hunter
Body = normal}
example area:
ZONE noneige
{
PosX = -100.388550
PosY = 438.709595
PosZ = -679.188782
HWidthX = 190.039322
HWidthY = 0.000000
HWidthZ = 0.000000
HLengthX = 0.000000
HLengthY = 150.349731
HLengthZ = 0.000000
HHeightX = 0.000000
HHeightY = 0.000000
HHeightZ = 145.658859}
Download the animation script pack for Agartha Dreamcast
Texture characters
Sick farmer
Hunter
Pope head
Ill gravedigger
Sick innkeeper?
Chief ill?
Waitress
Blacksmith's head
Innkeeper son
Stocky man
Stable's Head
Death Crunch Head
The menus (loading)
You can find the textures, sounds and more by downloading the pack below. (attention, the playable E3 prototype has not been reviewed yet)
Texture, Sound And Company Pack For Agartha (corrupted and demo2001 build)
The projets of research and Reverse Engineering undertaken on other games :
Feel free to look at " other canceled games " that I found. For the most curious of you, I have created a " list of all unreleased from theDdreamcast ".
Similar prototypes (Unreleased) : Agartha (DC) - Emulateur officiel Megadrive (DC) - Castlevania Resurrection (DC) - Half Life (DC) - Dalforce XOP (DC) - Flinstone (DC) - 4 x 4 Evolution PAL (DC) - Ring : L'Anneau des Nibelungen (DC) - Ecco 2 (DC) - Kyskrew (DC) - Propeller Arena (DC) - Geist Force (DC) - Scud Race Tech Demo Dreamcast - Shenmue 2 US (DC) - The Red Star (XBOX) - Heaven's Drive ( version japonaise de Burnout 1) pour PS2 - Jekyll and Hyde (DC) - The Grinch Jap (DC) - Worms Pinball (DC) - Quake 3 Arena version japonaise (DC) - Vectorman PS2
the information right now is subject to change. A download link of the textures (there are plenty of them) will be available once the reverse work is finished.