29

Getting Started with ProGuard [FREE]

 5 years ago
source link: https://www.tuicool.com/articles/hit/Fz2imiB
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

There’s been a recent increase in popularity of Internet of Things, DIY boards and entry-level devices. The consequence has been a step back from writing large apps in favor of smaller ones. ProGuard is here help keep apps as small as can be.

Smaller apps download, install and run faster. This is important for business. The more time spent onboarding, the higher the chance the user is going to abandon your app and try something else.

In this tutorial, you’ll create a simple app which reveals details about sloths! You’ll learn how to strip down your app size by making use of ProGuard – an app shrinking and obfuscation tool. The optimizations that ProGuard performs translate to a certain level of obfuscation, which can add a minimal layer of security to help prevent reverse engineering or tampering with your app.

In the process, you’ll learn:

  • How to use the APK Analyzer
  • How to use ProGuard rules
  • How to debug a ProGuard’ed app

Note : This tutorial assumes that you’re already familiar with the basics of Android development and Android Studio. If Android development is new to you, first read through our Beginning Android Development andKotlin for Android tutorials.

Getting Started

Download and unzip the materials for this tutorial using the Download Materials button at the top or bottom of this page. Open the starter project in Android Studio 3.1.4 or greater, then build and run to see the app you’ll be working with.

AppStart-281x500.png

Right now there’s a simple screen that lists the six species of sloths. The app doesn’t do very much. You’ll make it more exciting by adding the BubblePicker library . It presents items that float around the screen in bubbles. You’ll want to check your app size along the way. Luckily, there’s a tool for that. :]

Using the APK Analyzer

The APK Analyzer is a tool for inspecting your finalized app and what contributes to it’s size. It presents a view with a breakdown of your app’s file size. You can see what’s taking up the most space, as well as the total method and reference counts.

Launch the analyzer by selecting Build ▸ Analyze APK . It will open a dialog for your filesystem. If necessary, navigate to the debug folder SlothSanctuary-Starter/app/build/outputs/apk/debug . Select the app-debug.apk file and click OK to open the APK Analyzer. Note the file size of the current APK. You’ll use this tool several more times in this tutorial.

APKAnalyzer-1-650x166.png

Note : There are many other tools that engineers use to analyze apps, such as JD-GUI, APKTool, and Jadx. Some of these tools help reverse engineers decompile the app back to its original code. Sometimes it’s done for fun, but often reverse engineering is done for theft of intellectual property or app cloning.

Adding the BubblePicker Library

Open the build.gradle file for the app module and add the following to the list of dependencies:

implementation 'com.github.igalata:Bubble-Picker:v0.2.4'

Perform a Gradle sync , then select Build ▸ Make Project . Once finished, run the APK Analyzer again. You’ll notice adding the dependency added a few megabytes to the APK file without even writing any code! You won’t be using every part of this library – this is where ProGuard comes in. ProGuard will do its job to remove all the code that is not accessed by the app.

Enabling ProGuard

Enabling ProGuard is simple! In the build.gradle file for the app module, replace the buildTypes code with the following:

buildTypes {
  release {
    minifyEnabled true
    proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
  }
  debug {
    minifyEnabled true
    proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
  }
}

Setting minifyEnabled to true enables ProGuard. ProGuard looks at the entry points of your app and maps out the code that the app can reach. It removes the rest, and replaces the names of classes and methods with shorter ones, making for a much smaller APK size! Be aware that using ProGuard results in slower build times.

Even so, ProGuard often mistakenly obfuscates and removes code that it thinks you’re not using, even when you are . You’ll need to test often that everything still works with ProGuard enabled as you go along. The earlier you find problems in the build, the easier it will be to fix them. :]

Sync Gradle, then build and run. Notice that there are already compiler errors:

ProGuardWarnings1-650x145.png

The compiler errors include “Can’t find referenced class org.sl4j” and “Can’t find referenced class sun.misc.Unsafe” . The first way you’ll look to solving these ProGuard problems is through online research.

Head over to Bubble Picker library’s GitHub page to see if there’s any documentation about using the library with ProGuard. Sometimes the README page will have ProGuard information, but in this case it doesn’t. Next, select Issues .

Github1-650x275.png

In the search field, remove the is:open , add sl4j and press enter . Good luck – issue #61 looks like the same issue, with suggestions to add some “don’t warn” exceptions for ProGuard.

Github2.png

Adding “Don’t Warn” Rules

“Don’t warn” rules tell Android Studio to ignore warnings. This can be dangerous, but if you know for sure that you’re not using a part of the code, it can come in handy. They work by specifying the package name. * is a wildcard – it doesn’t include sub-packages, whereas ** includes sub-packages. The rules for ProGuard go into the proguard-rules.pro file.

Add the following to the end of the proguard-rules.pro file to ignore warnings for the org.slf4j package:

-dontwarn org.slf4j.**

The next warnings that all refer to sun.misc.Unsafe . rx.internal are in reference to the RxJava library. Head over to the RxJava Github page and click on Issues . Replace is:open with sun.misc.Unsafe and press enter . You’ll see a few posts about the same issue. In this post , it states that sun.misc is never used so it’s safe to ignore. Add the following in the proguard-rules.pro file, right under the line you just added:

-dontwarn sun.misc.**

Select Build ▸ Make Project . Now it builds successfully!

Note : You may see suggestions for “-dontwarn *” floating around on forums, but it’s very bad practice. This means don’t warn “all”. It will fix irrelevant warnings, but will also ignore critical ones where something is actually wrong.

Run your APK Analyzer again. You’ll notice the APK size is much smaller now. That’s because ProGuard has removed all of the code you’re not using. Now that your project builds, you can start to add in code to display the bubbles.

Adding BubblePicker Code

In the res/layout/activity_main.xml file, replace the second TextView ( descriptionTextView ) with the following:

<com.igalata.bubblepicker.rendering.BubblePicker
  android:id="@+id/picker"
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  app:backgroundColor="@android:color/white" />

This adds the BubblePicker to the main layout. In the MainActivity.kt file, add the following to the setupBubblePicker() method:

picker.bubbleSize = 50
picker.centerImmediately = true
picker.adapter = object : BubblePickerAdapter {

  val colors = resources.obtainTypedArray(R.array.colors)
  val titles = listOf("1", "2", "3", "4", "5", "6")

  val multiplier = 2
  val modulus = 8
  val addition = 1

  override val totalCount = titles.size // 1

  override fun getItem(position: Int): PickerItem { // 2
    return PickerItem().apply {
      title = titles[position]

      val start = colors.getColor((position * multiplier) % modulus,0)
      val end = colors.getColor((position * multiplier) % modulus + addition,0)
      gradient = BubbleGradient(start, end, BubbleGradient.VERTICAL)

      textColor = ContextCompat.getColor(this@MainActivity, android.R.color.white)
    }
  }

}

picker.listener = object : BubblePickerListener { // 3

  override fun onBubbleSelected(item: PickerItem) {

  }

  override fun onBubbleDeselected(item: PickerItem) {

  }

}

Note : This tutorial assumes you’re familiar with handling imports. If you don’t have on-the-fly imports set up, import by pressing option + return on a Mac or Alt + Enter on a PC while your cursor is over the item that needs to be imported.

Here’s what’s going on in the updated method:

  1. You are overriding totalCount to tell the BubblePicker how many bubbles there will be.
  2. You’re overriding getItem() to return a customized PickerItem .
  3. You set up the BubblePickerListener to handle selecting a bubble.

Finish off the implementation by adding this to the end of onResume :

picker.onResume()

Add this to the end of onPause :

picker.onPause()

Build and run the app. Uh oh — the app crashes with a NullPointerException ! Note that several methods in your stack trace are obfuscated – the names are changed and minified. This is one of the key features of ProGuard.

ProGuardCrash1-650x239.pngbasic-sad-1.png

Check the output log to narrow down what the problem is. You can see in the Run tab that it has something to do with “GL” and “onDrawFrame” in the BubblePicker library.

Note : It’s always good practice to add sufficient logging in your catch statements, nullability checks and error states. With ProGuard, this is crucial – in the event of a problem, it will help lead you or other developers to the root of the issue, especially when the method names in stack traces are obfuscated.

Debugging with ProGuard Output Files

When ProGuard finishes running, it produces 4 output files. They are:

  • usage.txt – Lists code that ProGuard removed.
  • dump.txt – Describes the structure of the class files in your APK.
  • seeds.txt – Lists the classes and members that were not obfuscated. Good to verify that you have obfuscated your important files.
  • mapping.txt – The file that maps the obfuscated names back to the original.

You can use the mapping file to find the culprit of the crash.

Run the APK Analyzer again, then select the classes.dex file. Your crash points to obfuscated code looking something like org.a.d.l.a . The single characters you see may vary from this example but you can follow the directory path of the characters:

ProGuardObfuscated.png

At first, it’s not clear what the directories are. Click the Load Proguard mappings… button to map the obfuscated names back to the original.

LoadProguardMappings.png

Select the mapping.txt file in the debug folder and click OK .

Toggle the Deobfuscate names button to the left of the Change ProGuard mappings… button to switch between obfuscated and deobfuscated code. You can trace the problem down to the jbox2d library.

ProGuardUnobfuscated-1.png

You need to preserve jbox2d so that ProGuard doesn’t muck with it.

There are a few more things you should know about the mappings file:

  • Every time you make a release build, the mapping.txt file is rewritten. That means you must save each copy with each release of your app. That way, when you receive an obfuscated stack trace for a particular app release, it will be useful.
  • You can upload your mapping.txt file to Google Play to deobfuscate your crash stack traces.
  • If you’re using Fabric , a deployment and crash reporting tool from Google, instructions for uploading your mapping.txt file are here .

Adding Keep Rules

Keep rules tell ProGuard not to obfuscate certain parts of your code. Some options are:

keep
keepclassmembers
keepclasseswithmembers

ProGuard rules are written in a specific template format . It’s best practice to use explicit keep rules, rather than keeping the entire class. Instead of preserving the entire BubblePicker library, you only need to keep org.jbox2d and its sub-packages. The format is the same as the dontwarn rules.

Add the following to the proguard-rules.pro file right after the dontwarn rules you added:

-keep class org.jbox2d.** { *; }

Inside the curly braces, you told ProGuard to match any method name.

Build and run the app. When your app loads up, you should see the bubbles floating around on the screen.

BubbleNumbers-281x500.png

Note : If you’re sharing your code, write keep rules as you write your code and be sure to publish them on your site or GitHub README page so other developers can easily use your code without any problems.

Now you’ll set up some real sloth data.

Adding Data to the BubblePicker

There’s a file included in the res folder of the project called sloths.xml . Since it’s in XML format, you’ll need a way to get that data into Kotlin objects. You’ll use another third party library included with the Retrofit package called SimpleXML .

Add this code to the list of dependencies in the build.gradle app module file. You can exclude groups and modules that you’re sure you won’t use:

implementation ('com.squareup.retrofit2:converter-simplexml:2.0.0-beta3'){
    exclude group: 'xpp3', module: 'xpp3'
    exclude group: 'stax', module: 'stax-api'
    exclude group: 'stax', module: 'stax'
}

Sync Gradle, then build and run the app.

You’ll have some new warnings: a reference to org.codehaus.mojo.animal_sniffer.IgnoreJRERequirement , and many variations of warnings about javax.xml.stream.** .

ProGuardWarnings2-650x269.png

You won’t be using Java’s XML stream feature. For this tutorial’s purpose, it’s safe to ignore the animal_sniffer warnings. Add the following to the proguard-rules.pro file:

-dontwarn javax.xml.stream.**
-dontwarn org.codehaus.mojo.animal_sniffer.IgnoreJRERequirement

Select Build ▸ Make Project . This time the build should succeed.

Open the SlothViewModel.kt file and replace the loadSloths() function with the following:

private fun loadSloths(resources: Resources) { // 1
  val serializer = Persister()
  val inputStream = resources.openRawResource(R.raw.sloths) // 2
  val sloths = serializer.read(Sloths::class.java, inputStream) // 3
  sloths.list?.let { theList ->
      val map = theList.associateBy( { it.name }, {it})
      this.sloths = map.toSortedMap()
  }
}

Here:

  1. You removed the explicit return type for the function.
  2. You opened the sloths XML file an an InputStream .
  3. You retrieved a list of Sloths by invoking read() .

The XML parser knows how to map the XML fields to your Kotlin objects by using annotations. In the Sloths.kt file, add this annotation above the Sloths constructor:

@Root(name = "sloths", strict = false)

This tells the parser to look for the root node called “sloths”.

Then add annotations above the val list property in the constructor:

@field:ElementList(entry = "sloth", inline = true)
@param:ElementList(entry = "sloth", inline = true)

@field and @param let the parser know to look for “sloth” items.

The same will need to be done for the Sloth class. In the Sloth.kt file, replace everything after the import statement with this:

@Root(name = "sloth", strict = false)
data class Sloth constructor(
  @field:Element(name = "name")
  @param:Element(name = "name")
  var name: String = "",

  @field:Element(name = "realName")
  @param:Element(name = "realName")
  var realName: String = "",

  @field:Element(name = "imageResource")
  @param:Element(name = "imageResource")
  var imageResourceName: String = "",

  @field:Element(name = "description")
  @param:Element(name = "description")
  var description: String = "") : Serializable

In the MainActivity.kt file, add a ViewModel variable right above onCreate() :

private val viewModel: SlothViewModel by lazy {
  ViewModelProviders.of(this).get(SlothViewModel::class.java)
}

Add a call to get the sloth data as the first line of setupBubblePicker() :

val map = viewModel.getSloths(resources)

Replace the line that reads val titles = listOf("1", "2", "3", "4", "5", "6") with the following:

val titles = map.toList()

And finally, replace the title = line with this:

title = titles[position].first

Build and run the app. Notice the NoSuchMethodException crash.

ProGuardCrash2-650x34.png

You know it must be the code you just added. Doing similar debugging to what you did earlier, you can narrow the problem down to the Parameter object inside SimpleXML. This time you’ll take a deeper look at the problem. SimpleXML works by loading XML entities presented at runtime, then instantiates Kotlin counterparts. Kotlin can only do this by using introspection and reflection .

Introspection and Reflection

Sloths hang out in trees for hours and hours, not seeming to do very much. It’s probably because they’re busy introspecting and reflecting on life. In Kotlin, introspection and reflection are features of the language that inspect objects and call methods dynamically at runtime.

ProGuard looks at the static version of your app, but doesn’t actually run your app, so it cannot know which methods are reachable using introspection and reflection. Remember that ProGuard takes long class names and replaces them with smaller names. If something tries to reference a name at runtime with a constant string, the name will have changed. You can often tell this is happening with the NoSuchMethodException .

You need to tell ProGuard to keep the sections that use reflection, like the @param code you added. You’ll use the implements keyword to keep all classes that extend or implement Parameter . Add the following to the end of the ProGuard rules file:

-keep class * implements org.simpleframework.xml.core.Parameter { public *; }

While you’re at it, go ahead and add a bit more functionality to the app. In the MainActivity.kt file, add this to the onBubbleSelected() method of the BubblePickerListener :

val showDetailsIntent = Intent(picker.context, SlothDetailActivity::class.java)
val pet = map[item.title]
showDetailsIntent.putExtra(SLOTH_KEY, pet)
startActivity(showDetailsIntent)

item.isSelected = false

Build and run the app. You should see the bubbles populated with the different species of sloths.

AppFinal-281x500.png

Tap on a bubble to reveal more information about each species.

AppDetails-281x500.png

Congratulations! You’ve set up a cool looking app using the default ProGuard settings.

basic-happy-3.png

You could stop here, but if you’re interested in how the app size can be further reduced, continue reading.

Enabling Advanced Optimizations

ProGuard provides an advanced optimization profile. It’s not used as the default because it can cause further build and runtime errors. You’ll enable the advanced optimizations by swapping the default proguard-android.txt file out with proguard-android-optimize.txt .

In the build.gradle app module file, replace the proguardFiles line in the debug section with the following:

proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'),
          'proguard-rules.pro'

Sync Gradle, then build and run the app. The build time will be much longer because ProGuard is performing more analysis and optimizations inside and across methods. It will seem very slow, much like a sloth climbing a tree.

Your app should crash with the following error:

ProGuardCrash3-1.png

Here, the @ symbol is in reference to an annotated class – in this case, classes annotated with org.simpleframework.xml.* . Add a rule to the end of the ProGuard rules file to keep methods annotated with org.simpleframework.xml.* :

-keepclassmembers class * { @org.simpleframework.xml.* *; }

This time, the error mentions a constructor instead of a missing class, which is why this code uses keepclassmembers to keep class members only.

Build and run the app. You should see your app back up and running again.

Note : This might be a good time to stop and check the APK Analyzer again to see how the size is doing.

You’ve set up a fancy ProGuard optimized app for your Sloth Sanctuary. The Sleuth Zoo is very competitive – they’ve been taking all your ideas and making a profit off of them. How evil! You think that when you release your app, they’re going to analyze your APK and steal your “secret sauce”. Since ProGuard performs obfuscation, this is yet another place that it can help you out.

Understanding Obfuscation

Sloths hide themselves at the tops of trees to prevent predators finding them from below. Many have a symbiotic relationship with green algae. It grows on their backs so that they blend in with the green leaves. This tricks eagles and other flying predators. Sloths obfuscate themselves, which really comes down to trickery and hiding.

Obfuscation is not encryption. Throwing ProGuard on your release can deter the casual attacker to move on. However, it’s not a substitute for proper security measures. You should never store sensitive API keys, tokens or passwords anywhere in the APK. Instead, have those items sent to the app encrypted upon authentication .

That being said, obfuscation is sometimes used to hide or obscure proprietary logic or a secret algorithm. Sometimes developers apply manual obfuscation. Some examples are string splitting, dummy code, disguising the names of methods, or using reflection to muddy the app flow. You’ll add some reflection code that obfuscates your secret bubble-gradient formula. As of right now, it’s not impossible for an attacker to find the numbers used to make the gradient.

In the APK Analyzer, select the classes.dex file. Navigate to comraywenderlichandroidslothsanctuary . Right-click on MainActivity$b and choose Show Bytecode . Integers e , f and g are the multiplier , modulus and addition variables of 2 , 8 and 1 you added earlier.

Obfuscation0-650x436.png

When Android Studio compiles your app, it puts the code into that classes.dex DEX (Dalvik Executable) file. DEX files contain Bytecode – an intermediary set of instructions that a Java Virtual Machine (JVM) runs or ART (The Android Runtime) later converts to native code. With this being exposed, an attacker could potentially see values that your variables hold!

You’ll now make it much harder to find those gradient numbers. In the ProGuard rules file, add rules that allow the use of reflection:

-keep class kotlin.reflect.jvm.internal.** { *; }
-keep class kotlin.Metadata { *; }

Under the hood, Kotlin classes contain metadata about an object. You’ll need to keep that metadata in order to use reflection.

Open the BN.kt file. Notice the class was manually obfuscated by abbreviating the class name and methods. The comments explain each abbreviation. Comments don’t make it into the APK.

Add the following code to the sv() method:

val kClass = Class.forName(ownerClassName).kotlin // 1
val instance = kClass.objectInstance ?: kClass.java.newInstance() // 2
val member = kClass.memberProperties.filterIsInstance<KMutableProperty<*>>()
    .firstOrNull { it.name == fieldName } // 3
member?.setter?.call(instance, value) // 4

Wait, what? What is this magic?

While you don’t need to understand it, this code does the following:

ownerClassName
fieldName
value

The real magic happens when the app invokes sn() (setupNumbers), which looks for the com.raywenderlich.android.slothsanctuary.GO class. It finds the fields named f1f3 and swaps the values out for something else at runtime.

To put this code to use, go back to the MainActivity.kt file and add this code to onCreate() , right before the setupBubblePicker() call:

val bn = BN() //BubbleNumbers
bn.sn() //bubbleNumbers.setupNumbers

Then find the lines that declare multiplier , modulus and addition in setupBubblePicker() and replace them with this:

val multiplier = GO.f1 //GradientObject.field1
val modulus = GO.f2 //GradientObject.field2
val addition = GO.f3 //GradientObject.field3

Note : Sometimes attackers also reverse engineer apps in hopes of patching or hooking security checks out of the code. A good example is when a feature is only available with a paid subscription or after a user achieves a level in a game. It’s recommended to do those types of checks on a server. ProGuard can still help by obfuscating the code that makes the request to the server.

This is an example of using the Kotlin reflection library to call methods by name, but now ProGuard is changing those names. Build and run the app to get the ClassNotFoundException once again.

ProGuardCrash4.png

You can use keep rules again to preserve the class, but this time around you’ll do it by using @Keep annotations.

Adding Annotations

With the Annotations Support Library , you can add @Keep to methods and classes you want to preserve. This is a great feature because it acts like documentation. The ProGuard information sits above your method as opposed to being in a separate file. Adding @Keep on a class will preserve the entire class. Adding @Keep on a method or field will keep the name of the method or field as-is.

In the BN.kt file, add a @Keep annotation to the top of the GO object.

Build and run the app. You should now see the bubbles again. This time, if you analyze the APK, you’ll see the secret gradient numbers are not so obvious:

Obfuscation1a-590x500.png

Even if you follow those methods to the GO companion object, the numbers that are apparently returned are not the real ones. You tricked em!

Obfuscation2-650x331.png

You’re becoming a pro at ProGuard!

Where To Go From Here

You survived a crash course on ProGuard. Download the final project using the Download Materials button at the top or bottom of this tutorial.

You looked at how ProGuard has problems with reflection. ProGuard also has issues when you call a method from JNI (Java Native Interface). If you’re working with JNI, check out solutions in the JNI training article .

An Android Library (AAR) is able to make use of a transparent method that retrieves published keep rules automatically. See consumerProguardFiles on how to take advantage of this.

The makers of ProGuard, GuardSquare , have a commercial solution called DexGuard . It offers more protection than obfuscation. DexGuard encrypts the classes and strings as well as assets and resource files. It also provides app integrity checking and certificate pinning.

While Android Studio comes with ProGuard built in, Google has been developing R8, which aims to be a drop-in replacement. It’s a new code shrinking and obfuscation tool that could replace ProGuard in the future. Check out Google’s R8 page for more information.

If you have any questions or comments, please join the discussion below!


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK