Once you have extracted the program to a folder, simply clicking on the exe causes a console window to quickly open and close. That's normal.
Normally, to use AtlusScriptCompiler you have to use commandline arguments. If you have never used the Windows command prompt before, don't worry. It's quite simple:
You can open the command prompt one of several ways:
Searching for "cmd" in the Windows search bar
Pressing ctrl + R and typing "cmd" in the Run window
Typing "cmd" in the address bar in Windows explorer
Right click inside a folder in Windows Explorer while holding shift and choose "Open Command Window here" (might have to do this first on Windows 10)
Once the command prompt has opened, all you have to do is type the path to the EXE (i.e. "C:\User\Username\Desktop\AtlusScriptCompiler\AtlusScriptCompiler.exe"). It's smart to wrap the path in quotation marks in case it contains spaces.
Note that you can also drag the EXE onto the command prompt window to automatically fill in the path.
If you used one of the latter two methods of opening the command prompt, if the folder contains AtlusScriptCompiler.exe, all you have to type is "AtlusScriptCompiler.exe".
Press enter, and the program will print information to the console without immediately closing.
You can press up on the keyboard to copy what you previously typed in order to make changes.
Supplying the Input File
Whether the file you want to decompile is a .bf or .bmd, if you supply the path immediately after "AtlusScriptCompiler.exe" (followed by a space) it will be used as the input file.
If your input is a .flow or .msg, you may be looking for instructions on compiling instead.
Supplying the Decompile Command
Follow up the command so far with another space and then "-Decompile" in order to tell the program what to do with the input file.
Supplying the Library
You can skip this step if you're decompiling a BMD. For AtlusScriptCompiler to successfully resolve .bf to .flow, a library must exist that defines the game's functions and their parameters. In order to create a library, function signatures must be dumped from the game using a script like this one.
"-Library P5" (if your game is Persona 5)
"-Library P4" (if your game is Persona 4)
"-Library P4G" (if your game is Persona 4 Golden)
"-Library P3" (if your game is Persona 3)
"-Library P3F" (if your game is Persona 3 FES)
"-Library PQ2" (if your game is Persona Q2)
"-Library SMT3" (if your game is SMT Nocturne)
"-Library DDS" (if your game is Digital Devil Saga)
The script will fail to decompile if the appropriate library isn't present.
At the time of writing, there is no library for Persona 3 Portable.
Supplying the Encoding
For AtlusScriptCompiler to successfully decompile to either .flow or .msg, encoding must be specified.
"-Encoding P5" (if your game is Persona 5)
"-Encoding P4" (if your game is Persona 4)
"-Encoding P3" (if your game is Persona 3)
"-Encoding SJ" (if your game is Persona Q(2) or uses Shift-JIS/CP932)
Specifying the Output File
While the command so far is enough to output a file, it would be automatically named after the input. For instance, field.bmd would generate field.bmd.msg.
If you would rather name it something else like field2.msg, or replace field.msg, try something like the following:
Once again, remember to wrap paths to files in quotation marks in case they contain spaces.
Using the GUI Frontend
There is an unofficial GUI for AtlusScriptCompiler that allows you to simply drag and drop files to compile or decompile them.
You can get it from here. Just extract the contents of the .zip to the folder containing AtlusScriptCompiler.exe on your computer and run it.
Be sure to set the proper game in the dropdown first. If no output file is produced, check the "AtlusScriptCompiler.log" file in the same directory.
If a script has failed to decompile, an error message should be shown. It will usually display a description of what's wrong.
For a more visual guide on using AtlusScriptCompiler, see the video: