Agartha Dreamcast reverse engineering

Special thanks to:

  • Sifting of the forum "Obscure gamers"

To read on a PC:

Now most of the work I've done so far on it is just preliminary. To start off, I'm focusing on the April 18th release, which may be found here . The first obstacle was just getting the image mounted on my Linux machine. CDI images are proprietary, so usually I just convert them into an iso file and mount will be happy, but these images would simply not convert using any software that I knew. I finally managed to get the contents open using gditools , which I think is written by members of this forum? if so - thanks!

So on the filesystem root we have a bunch of files:

rock_snow_2.pvr.png
glyphe2.pvr.png

256344068 Apr 17 2001 000DUMMY.DAT

256344068 Apr 17 2001 0AGARTHA_DEMO2001.MPG

1105126 Apr 17 2001 1AGARTHA.BIN

3458 Mar 9 2001 ADXLISTDEMO.LST

36019 Apr 17 2001 AGARTHA.LST

13144526 Apr 17 2001 AGARTHA.PAK

18608 May 18 2000 AUDIO64.DRV

1021590 Mar 8 2001 BODNATH_EXTR01.ADX

1364634 Mar 8 2001 BODNATH_EXTR02.ADX

1311174 Feb 28 2001 BODNATH_EXTR08.ADX

1616130 Feb 28 2001 BODNATH_EXTR09.ADX

2110518 Feb 28 2001 GOKYO_EXTR01.ADX

1170810 Feb 28 2001 GOKYO_EXTR07.ADX

3225258 Feb 28 2001 GOSAINKUND_MOD_NIV_SHORT.ADX

1943910 Feb 28 2001 KALAPATHAR_EXTR01.ADX

1553346 Feb 28 2001 KALAPATHAR_EXTR03.ADX

992394 Feb 28 2001 KALIGANDAKI_EXTR03.ADX

1037214 Feb 28 2001 KALIGANDAKI_EXTR07.ADX

1254042 Feb 28 2001 KALIGANDAKI_EXTR09.ADX

1122066 Feb 28 2001 KALIGANDAKI_EXTR11.ADX

5826294 Feb 28 2001 KANCHENJUNGA_EQ.ADX

1533150 Feb 28 2001 KANCHENJUNGA_EXTR09.ADX

1560114 Feb 28 2001 KANCHENJUNGA_EXTR11.ADX

1130742 Feb 28 2001 KANCHENJUNGA_EXTR13.ADX

2034630 Feb 28 2001 KIANGJING_EXTR03.ADX

219474 Feb 28 2001 KIANGJING_EXTR06.ADX

1721718 Feb 28 2001 KIANGJING_EXTR07.ADX

860022 Feb 28 2001 LADAKH_EXTR02.ADX

2196882 Mar 9 2001 LOUPIOTV3.ADX

2159300 Feb 26 2001 MOULIN.F3D

1739538 Feb 28 2001 PE03_22_EXTR01.ADX

4096 May 15 18:24 RESS

Coded:

//Agartha.PAK template

LittleEndian ();

// one per entry in AGARTHA.LST

SetBackColor (cPurple);

const uint NUM_FILES = 550;

uint offsets [NUM_FILES];

// before each file

struct File

{

uint32 uncompressed;

uint32 compressed;

uint16 flags;

byte data [compressed];

};

local uint i = 0;

while (i <NUM_FILES)

{

FSeek (offsets [i]);

if (i% 2! = 0) SetBackColor (cLtPurple);

else SetBackColor (cDkPurple);

File file;

i ++;

}

Agartha reverse hexe.png

The first interesting bit is the .mpg file, which is a 15 minute long video of the game, which is posted on @Sega Dreamcast Info 's site. The 000DUMMY.DAT is the same file. I guess it's there to pad out the disk. Weird, but I've seen weirder. The ADX files are all music tracks, best as I can tell. They are playable using VLC or ffplay, but some seem broken or unsupported. The RESS directory is curiously sparse, so this means the assets are hiding somewhere ... but where?

AGARTHA.PAK, of course.

However, when I popped it open in my hex editor it was immediately apparent this would be a more difficult task. The PAK file structure has no file manifest or anything, it's essentially just a big blob of data, yet there must be some way of mapping files to offsets somewhere. That's when I looked at AGARTHA.LST.

AGARTHA.LST is a big text file, with a bunch of file names inside it, one per line. This got me thinking, so I went back to my hex editor and sure enough, at the top of the file was a big blob of uints, and more interestingly, their values were all monotonic. I started jumping around inside the PAK file, treating each uint as an offset; sure enough, at each location was what appeared to be different files, and in the case of the PVR textures, their headers all lined up with the file names. This was the first important discovery in the journey of unlocking this treasure chest: The AGARTHA.LST contains a list of all files inside the AGARTHA.PAK; each file is written inside the PAK in the order it appears in the LST. There are 550 files total.

How ever, there's another big snag: the files are all compressed in what appears to be some form of LZW, or more probably, LZSS encoding. The give away here is that the early parts of the file are quite legible, but by the end of each file it's a binary mess. So in order to continue with the task of reversing Agartha, I have to figure out the decompression scheme. fortunately, LZW / LZSS isn't exactly black magic, but there are a ton of variations and at the moment I'm unsure of how to continue, so I have to ask: does anyone got any tips on reversing compression algorithms?

I gave it a try, and about half of the ADX files still fail to convert. I popped them open in the hex editor and it looks like the ones that fail are completely different formats all together. I looked up the ADX format to be sure, and they definitely do not match any known version of it. I did find something that might be interesting, though, see attached image. The valid ADX files do not have this section, so I wonder what's going on here.

I'll try a hand at cracking the decompression this weekend. It's going to feel great once we can get this treasure chest opened up.

Alright, I sat down and wrote out a script to dump the PAK file. So far it recreates the file structure and dumps the raw binaries to disk, but the compression remains a work in the progress. I did however learn that files are stored in at least 3 different modes, one of which is uncompressed. It's unclear if this specifies a compression quality setting or different algorithm at the moment.

Here's the script for anyone interested in examining the raw data:

planche9_256.pvr.png
planche8.pvr.png
corbac2.pvr.png
palbotte.pvr.png

Python:

#! / usr / bin / env python3

from struct import unpack, calcsize

import os

def uncompress (data):

return data

def main ():

PREFIX = 'AGARTHA'

DEST = 'content'

#Load the manifest

with open (PREFIX + '.LST' , 'rb' ) as f:

count = 0

lines = f.readlines ()

#Build a list of files

files = []

for ln in lines:

#Remove blanks and comments

ln = ln.decode ( 'latin' ) .strip () .lower () .replace ( '/' , '\\' )

yew '' == ln:

keep on going

yew '#' == ln [: 1 ]:

keep on going

#Insert the path into the list for later

files.append (ln)

count + = 1

#Figure out the common prefix

pref = os.path.commonprefix (files) .replace ( '\\' , '/' )

print ( f'Common Prefix: " {pref} " ' )

#Build a path list ...

paths = []

for p in files:

sanitised = os.path.normpath (p.replace ( '\\' , '/' ))

paths.append (sanitised.replace (pref, '' ))

#Extract files from archive ...

with open (PREFIX + '.PAK' , 'rb' ) as f:

offsets = unpack ( f '< {count} I' , f.read (calcsize ( f '< {count} I' )))

for i in range (count):

#Rebuild path on disk

fn = os.path.basename (paths [i])

dn = os.path.join (DEST, os.path.dirname (paths [i]))

os.makedirs (dn, exist_ok = True )

#Seek to find and read header

f.seek (offsets [i])

uncompressed, compressed, mode = unpack ( '<IIH' , f.read (calcsize ( '<IIH' )))

#Read and uncompress contents as needed

rate = 100 * compressed / uncompressed

MODES = [ 'uncompressed' , 'UNK0' , 'UNK1' ]

print ( f'Uncompressing " {paths [i]} ", ratio: {rate: .4} % ( {MODES [mode]} ) ' )

data = uncompress (f.read (compressed))

#Write it to disk and free data

with open (os.path.join (dn, fn), 'wb' ) as out:

out.write (data)

del data

if __name__ == "__main__" :

main ()

Addendum:

The encoding is definitely LZSS, at least the 0x02 mode. By complete serendipity I found out that it follows a similar scheme to the algorithm used in Allegro, where a control byte is issued for every 8 tokens, where a token may be a 1 byte literal or a 2 byte base / length pair. So at most there may be 16 bytes of data between control bytes. I haven't quite got it working though.

Addenum II:

I wrote out a routine to decompress the LZSS stuff as described above. It works for files encoded using Allegro, but only works partially for Agartha, so this seems to confirm to me that Agartha uses a variant. One observation I noticed is that I have to flip the byte order of the base / length pairs to get the proper number of bytes to decode, but I haven't quite managed to exactly get the proper bytes out of it. Single byte literals decompress perfectly, so there's no question about the over all scheme being similar to Allegro. It's just a matter of how to interpret the base index ...

ADXLISTDEMO.LST

To the best of my knowledge this is just a text file containing a list of music tracks, the same ones sitting on the disk root. One track per line, windows style line endings. I use this one to test since it's easy to tell when you get something right or wrong.

Unfortunately it seems to be more than just an endian issue, but it's too early to rule it out completely.

As an aside, I want to point out how perfect this algorithm is for the Dreamcast. For those unaware, the GD rom drive is quite slow and painful, and iirc, it's connected to the main cpu over the dainty G1 bus, so bandwidth is at a premium, but more than that, the DC's main cpu has a special feature that lets the programmer turn half the cache into super fast general purpose memory, which is perfect for this algorithm, which only needs a ring buffer and a few variables to keep track of its state. The Agartha team knew what was up!

Coded:

Ress\Music\Demo2001\bodnath_extr08.wav Ress\Music\Demo2001\bodnath_extr02.wav Ress\Music\Demo2001\kiangjing_extr08.wav Ress\Music\Demo2001\gokyo_extr04.wav Ress\Music\Demo2001\bodnath_extr11.wav Ress\Music\Demo2001\bodnath_extr12.wav Ress\Music\Demo2001\bodnath_extr10.wav Ress\Music\Demo2001\bodnath_extr05.wav Ress\Music\Demo2001\bodnath_extr06.wav Ress\Music\Demo2001\bodnath_extr04.wav Ress\Music\Demo2001\bodnath_extr13.wav Ress\Music\Demo2001\bodnath_extr14.wav Ress\Music\Demo2001\bodnath_extr09.wav Ress\Music\Demo2001\bodnath_extr07.wav Ress\Music\Demo2001\bodnath.wav Ress\Music\Demo2001\bodnath_extr01.wav Ress\Music\Demo2001\bodnath_extr03.wav Ress\Music\Demo2001\gokyo_extr07.wav Ress\Music\Demo2001\gokyo_extr06.wav Ress\Music\Demo2001\gokyo_extr05.wav Ress\Music\Demo2001\gokyo_extr08.wav Ress\Music\Demo2001\gokyo.wav Ress\Music\Demo2001\gokyo_extr01.wav Ress\Music\Demo2001\gokyo_extr02.wav Ress\Music\Demo2001\gokyo_extr03.wav Ress\Music\Demo2001\Gosainkund_mod_niv_short.wav Ress\Music\Demo2001\kalapathar.wav Ress\Music\Demo2001\kaligandaki.wav Ress\Music\Demo2001\kalapathar_extr05.wav Ress\Music\Demo2001\kanchenjunga_extr11.wav Ress\Music\Demo2001\kanchenjunga_extr15.wav Ress\Music\Demo2001\kanchenjunga_extr05.wav Ress\Music\Demo2001\kanchenjunga_extr09.wav Ress\Music\Demo2001\kanchenjunga_extr12.wav Ress\Music\Demo2001\kanchenjunga_extr16.wav Ress\Music\Demo2001\kanchenjunga_extr06.wav Ress\Music\Demo2001\kanchenjunga_extr13.wav Ress\Music\Demo2001\kanchenjunga_extr07.wav Ress\Music\Demo2001\kanchenjunga_eq.wav Ress\Music\Demo2001\kanchenjunga_extr10.wav Ress\Music\Demo2001\kanchenjunga_extr14.wav Ress\Music\Demo2001\kanchenjunga_extr08.wav Ress\Music\Demo2001\kaligandaki_extr12.wav Ress\Music\Demo2001\kaligandaki_extr13.wav Ress\Music\Demo2001\kaligandaki_extr10.wav Ress\Music\Demo2001\kaligandaki_extr11.wav Ress\Music\Demo2001\kaligandaki_extr06.wav Ress\Music\Demo2001\kaligandaki_extr07.wav Ress\Music\Demo2001\kaligandaki_extr05.wav Ress\Music\Demo2001\kaligandaki_extr14.wav Ress\Music\Demo2001\kaligandaki_extr15.wav Ress\Music\Demo2001\kaligandaki_extr08.wav Ress\Music\Demo2001\kaligandaki_extr09.wav Ress\Music\Demo2001\kalapathar_extr01.wav Ress\Music\Demo2001\kalapathar_extr02.wav Ress\Music\Demo2001\kalapathar_extr03.wav Ress\Music\Demo2001\kalapathar_extr04.wav Ress\Music\Demo2001\kaligandaki_extr01.wav Ress\Music\Demo2001\kaligandaki_extr02.wav Ress\Music\Demo2001\kaligandaki_extr03.wav Ress\Music\Demo2001\kaligandaki_extr04.wav Ress\Music\Demo2001\kanchenjunga_extr01.wav Ress\Music\Demo2001\kanchenjunga_extr02.wav Ress\Music\Demo2001\kanchenjunga_extr03.wav Ress\Music\Demo2001\kanchenjunga_extr04.wav Ress\Music\Demo2001\kiangjing_extr10.wav Ress\Music\Demo2001\kiangjing.wav Ress\Music\Demo2001\kiangjing_extr04.wav Ress\Music\Demo2001\kiangjing_extr11.wav Ress\Music\Demo2001\kiangjing_extr01.wav Ress\Music\Demo2001\kiangjing_extr02.wav Ress\Music\Demo2001\kiangjing_extr03.wav Ress\Music\Demo2001\kiangjing_extr07.wav Ress\Music\Demo2001\kiangjing_extr05.wav Ress\Music\Demo2001\kiangjing_extr06.wav Ress\Music\Demo2001\kiangjing_extr09.wav Ress\Music\Demo2001\ladakh.wav Ress\Music\Demo2001\ladakh_extr03.wav Ress\Music\Demo2001\ladakh_extr01.wav Ress\Music\Demo2001\ladakh_extr02.wav Ress\Music\Demo2001\Patshupatinath_modif_niv.wav Ress\Music\Demo2001\Pe03_22_extr01.wav Ress\Music\Demo2001\loupiotV3.wav

I figured it out.

Here's the trick:
the pairs are 16 bits wide, written little endian; the lower 4 bits are the length - 3, and the remaining 12 are the unsigned base. when you decode the pair, subtract base + 1 from the ring buffer position and cache it. this will be your offset, now just read out length bytes from the ring buffer relative to the offset.

I ran some metrics on the files, and about 60% are encoded in LZSS, 30% in some other encoding - probably LZW - and 10% are uncompressed. Here are a few bits that I've found so far. I've employed the PVR routines that I wrote for Castlevania again, so we can see the textures ...

A lot of Agartha appears to be scriptable in plain text, very similar to how Castlevania was. Again, this is probably because DeLoura's book was the hot read back then!

Coded:

/***************************************************************************** ______________________________________________________________________ ) ) / Agartha - Demo2001 / / Playable.def / (_____________________________________________________________________( | ) | | Purpose / Palyable demo. Player can walk through Agartha world | | ( as well as meet some characters... |==========\==========================================================| | ) | | Author / Sébastien Viannay - NoCliché | |=========(===========================================================| | \ | | History ) 01/03/2001 : Creation. | |__________/__________________________________________________________| *****************************************************************************

/ // include the entry point scene for the playabale demo part

INCLUDE "Village\Village.def"

Some sounds:

00:00 / 00:01
00:00 / 00:01
00:00 / 00:00
00:00 / 00:03
00:00 / 00:05
00:00 / 00:01
00:00 / 00:08
00:00 / 00:03
00:00 / 00:03

Kirk (the hero) texture:

Kirk's hair

kirk_cheveux Agartha Dreamcast

Kirk's head

kirk_tete Agartha Dreamcast

Kirk's jacket

kirk bag

kirk_sac Agartha Dreamcast
kirk_veste Agartha Dreamcast
chien256x256.pvr.png
plateau en or.pvr.png
sauer.pvr.png

Book (Lore)

Page 1

 

In the beginning, the massive earthquake, that the Bible erroneously calls Flood,

deeply transformed terrestrial geography and led the 13 tribes on the path of great migrations.

Guided by those that the Pnakotics Manuscrits (Vatican libraries archives) have designed under the generic name of "the 9 unknown ones", those tribes populated the highlands that escaped the rising waters.

That's how various cyclopean cities were born. Ancient papyrus preserved in the archeological museum of Cairo and bearing the seal of Meneptah (1200 years before our era) mention for example the forbidden city of the highlands of Leng which has seemingly entirely vanished following the cataclysm that shook the region of Pamir.

Page 2

 

It was during risky peregrinations, that led by Hassuna, the Sennacherib clan, also called children of the first world, discovered the gigantic schist grottoes that formed organically not far from the higher spring of the Brahmaputra (Himalayan Chain), at 6700 meters above the current sea level.

The Sennacherib decided to settle and take root there. One can find multiple allusion to this fact in the Mahabharata (Third Book)

That's probably how an underground passage was found, linked to a complex network of passageways leading to the earth core, legendary location in multiple mythologies that refer to what we know today as the Hollow Earth.

Page 3

 

Following the guidance of the "Enlightened One", the clan decided after the ritual sacrifices to build a secret city that could shelter a part of the human species should a cataclysmic event happen. No one knows exactly what techniques were used to accomplish this "Grand Design", or Agartha in Tibetan, which also signifies:

The subterranean kingdom at the center of the Earth where the King of the World reigns.

 

The building of the city was spread over numerous generations, and it seems, based on the few documents that have survived, that Agartha was never completely built. Local folklore (such as the chantings of Lapcha) attributes to the "inevitable devastation", this frenzy that led to enslavement and death a lot of the builders of this subterranean metropolis. Although it seems that events far more dramatic than those prophecies preceded this titanic work.

Page 4

 

Agartha was the home of powerful malevolent forces fallen and banished from the divine kingdom. Under the divine curse prohibiting them from appearing in the open, those infernal forces have been buried in the deepest part of the earth.

Planning their return on Earth, they started to decimate the children of the first world using an unknown disease. By doing this they made Agartha, an antechamber to hell in which they reign supreme.

Leading the demoniac hordes, is the one they call The Sentinel (Y'aga Heer'GHta in the Sennacherib dialect). If the writings are to be trusted, the city of Agartha communicates and opens towards other heavens (please refer to the works of Julius Horbiger, Bale faculty 1724 and especially the corpus 46 entitled "On the Hollow Earth")

Page 5

 

Indeed it seems that the children of the first world had planned to return to the surface after the “inevitable devastation” that they feared. Survivors of the epidemic were able to transmit to other people some of their knowledge. That's why Gozer's tablets and “Dead Sea Scrolls” allude to the existence of a “relay that leads soul from the surface to the depths below”.

Once a century, when the Scorpio enters the Virgo square, a chosen woman is born. A direct descendent of the Sennacherib family, she represents the ultimate salvation or the ultimate threat that can either save or doom the land. Indeed, with the chosen one's sacrifice, carried out under certain conditions, the Sentinel can either rise in broad daylight or be cast back to its lair.

Nowadays, the only custodians of this knowledge and undertakers of this task, have gathered in a secret society, called the “Order of the Sennacherib”. Without being descendents of the clans, their mission is to fight Evil. Their sworn enemies, hell bent on awakening the Sentinel, are called “the Conspirators of Twilight”

English translation by Vince

Page 1

Page 2

Page 3

Page 4

Page 5

Crawley book

Texture characters

Sick farmer

fermier malade Agartha Dreamcast

Hunter

chasseur Agartha Dreamcast

Pope head

pope_tete3 Agartha Dreamcast

Ill gravedigger

fossoyeurmalade Agartha Dreamcast

Sick innkeeper?

aubergiste_ialpha Agartha Dreamcast
chef_i Agartha Dreamcast

Chief ill?

Waitress

serveuse Agartha Dreamcast

Blacksmith's head

teteforgeron Agartha Dreamcast

Innkeeper son

aubergiste_fils_alpha.pvr.png

Stocky man

texturehommetrapus alpha Agartha Dreamcast

Stable's Head

paltete Agartha Dreamcast

Death Crunch Head

croqu_tete Agartha Dreamcast

The menus (loading)

loading1 Agartha Dreamcast
loading7 Agartha Dreamcast
loading2 Agartha Dreamcast
loading9 Agartha Dreamcast
loading6 Agartha Dreamcast
loading5 Agartha Dreamcast
loading8 Agartha Dreamcast
loading3 Agartha Dreamcast
loading4 Agartha Dreamcast
loading11 Agartha Dreamcast
loading10 Agartha Dreamcast

the information right now is subject to change. A download link of the textures (there are plenty of them) will be available once the reverse work is finished.