You can download music to your phone without having to stream it over your data connection, the app isn't fully offline and requires a data connection to download the list of tracks within an album and verify the tracks are still available.
To be able to download music you must add it to My Music, you can do this for albums or artists by tapping on the + icon on the bottom toolbar if its available. Then return to the home page of the app and goto My Music. In there you will then have the option to download your music.
If the music is Amazon Prime music then downloading can only occur while the app is running, so please wait while this download completes before doing anything else with your phone. In the future it might be possible to download in the background but not at the moment.
If the music is tracks you have uploaded to Amazon or have purchased, then you can download this music in the background and continue to use your phone whilst the downloads happen. If your battery is low or battery saver is turned on then these background downloads won't complete until you charge your phone. It may also be possible to configure your phone so that background downloads only occur over Wifi, if that is the case then you will require a Wifi connection to download the tracks.
Once the tracks have downloaded, when you view an album or playlist the tracks will show downloaded instead of stream. Then you can be sure that you won't incur any data charges whilst playing these tracks. In the future it will be possible to view all your downloaded tracks/albums/etc but not currently.