5 dimensions of my wedding: Friendship

Two years ago, when I started the planning of my wedding with my wife-to-be (Carol), I thought it would be great if I compose the songs for the ceremony. When I first said that, most people did not really believe, I guess. But I actually did! Five songs that I call… ‘Five dimensions of my wedding’:

  • Friendship
  • Family
  • Candor
  • Love
  • Joy

I have spent a lot of my spare-time on this project, composing and producing the songs on the piano and on the computer. As a complete amateur I want to share with you the way I did it and hope you get inspired to do something similar! I plan to do a series of blog posts telling my experience as a musician, giving tips and ideas.

This first blog post is for the first song I composed for my wedding with Carol: Friendship! First check the final result: Friendship was composed for our friends. We used it for the entrance of our groomsmen and bridesmaids.

If you enjoyed the song, you can have it for free! It is possible to download the song from SoundCloud. You can also download the sheet on my MusicaNeo website.

My first worries were related to the main themes of the songs. How can I compose something that reflects what I feel about my friends? My answer was: ‘Piano Brainstorming’. Here is the recipe:

  1. Warm up your fingers
  2. Sit on the piano
  3. Think about your friends and good moment you had with them. Think hard and think sincerely (close your eyes if necessary).
  4. Put your cellphone to record everything by your side
  5. Start playing something on the piano. Something bizarre if necessary, but start something!!
  6. Don’t stop. Just keep playing varied stuff that comes out of your mind.
  7. Keep playing for several minutes until your mind is fully empty.
  8. Listen to the recording and replay the parts you loved.

You can do several piano brainstorms throughout some days. Then, you should be able to select the best themes you got and work on them, like polishing your precious piece. After that, you make a first recording on the computer. I used Apple’s Logic Pro X to record and edit my songs. I have chosen Logic because I found it very simple to use when compared to other software. Using Garage Band is a good free alternative, and there are other nice software you can use, like Sonar, Acid or Ableton Live. The advantage of buying a song production software is that you have much more control of your song than with free alternatives like Garage Band. You have good results with Garage Band, but tools like Logic make your songs more professional.

I connected my YAMAHA P-85 digital piano on my Mac using YAMAHA UX16 MIDI-USB original cable. Using the UX16 is essential if you plan to record your sustain pedal. I have the original LP5A 3-pedal unit for my P-85 and only UX16 were able to send the correct MIDI signals to the computer. If I try to use other cables, every time I press a pedal, the computer plays a note :-\. After the cable setup, recording on Logic was very easy. The Grand Piano samples that comes with Logic is just fine for a start. I recorded my overall idea for the Friendship song, like the sketch, using the metronome. Metronome makes things easier if you plan to manually edit your song later, so I strongly recommend it unless you need to do something very different in terms of tempo.

After recording the piano sketch, I decided to record the Strings. The problem is that the Violins samples that comes with Logic are terrible! I tried many tricks, tips from forums and Youtube videos, but the results were frustrating. I found the result just not realistic, very artificial. After doing a lot of research, I bought the Cinematic Strings 2 library (CS2) to produce more realistic strings solo. Cinematic Strings 2 may not be the best sample library in the market, but I had limited funds, so I found that CS2 was the best cost-benefit  for me.

In order to use third party libraries (such as CS2) within Logic Pro, you  need to use a software sampler such as KONTAKT player. It is totally free after you register on their website. Today I think It would have been better to buy Kontakt 5 in order to have sample editing advanced features. However, using the free edition worked fine (and cheaper) for my purposes. In order to use KONTAKT in Logic, open the mixer (press the X key on your keyboard), click on the Input button and browse to AU Instruments > Native Instruments > Kontakt 5, like shown on the Figure below:

Captura de Tela 2015-06-04 às 16.11.23

Using CS2 library really changed my motivation positively. I got very excited with the results and put a lot of strings on my first composition (maybe too much strings :P). I used tips from Youtube to make CS2 look even more realistic. Editing the modulation and volume like explained by this tutorial is essential to produce realistic strings arrangements.  You need a lot of patience and always remember that sometimes less is more. Avoid putting too much sounds together inadvertently.

As an example, in Friendship, I achieved good results customizing the modulation in the part between 0m:50s and 0m:56s of the song. You can hear this part of the song on the SoundCloud link in the beginning of the post. My intention was to force the sensation of breathing between two phrases of the song. To edit the modulation, press E on you keyboard to open the Piano Roll. Then follow the instructions of the pic below.

Captura_de_tela_04_06_15_16_45

This strategy worked fine, but only when the strings are playing on lower pitch registries. The same strategy did not worked very well with higher pitch registries. Another thing I tried to customize was the vibrato. CS2 automation for vibrato can be done by adjusting the Breath control in Logic. Try changing the controller to Breath and increase it on long notes to see the effect of the vibrato, like in the following picture:

Captura_de_tela_04_06_15_16_51

You can hear this part of the song between 1m:54s and 2m:00s. Producing realistic vibratos also depends on tuning Modulation and Volume. So, spend some time playing with those settings. My main critic regarding CS2 is the lack of more vibrato control. The vibrato amplitude in CS2 is limited. However, the library is well balanced and generally produces good results for you.

That’s it, hope you enjoyed!

GemFire In-Memory Map-Reduce with Java 8

When I started studying Gemfire, I decided I needed some hands-on experience with it. I enjoy learning by experience, thus I decided to share my own experience with you. Gemfire is an in-memory distributed database and it supports both relational-based store (Gemfire XD) and object store (key-value hash maps, Gemfire). On my first hands-on I started with the object store approach. Gemfire works great for data streams and it has a good distributed cache mechanism, which is very easy to use. I used vmware vFabric documentation and also this quickstart.

I started creating my IDE project. Using Maven, I only needed to add the repository on my pom file:

<repositories>
     <repository>
          <id>gemfire-repository</id>
          <name>Gemfire Repository</name>
          <url>http://dist.gemstone.com/maven/release</url>
     </repository>
 </repositories>
<dependencies>
     <dependency>
          <groupId>com.gemstone.gemfire</groupId>
          <artifactId>gemfire</artifactId>
          <version>7.0.2</version>
     </dependency>
</dependencies>

Then, Maven takes care of downloading necessary Jar libraries and etc. Creating a Gemfire cache is quite simple. First you create a cache:


import com.gemstone.gemfire.cache.Cache;
import com.gemstone.gemfire.cache.CacheFactory;
import com.gemstone.gemfire.cache.Region;

public class TwitterCache {

private final Cache cache;
private final Region<Long, Status> tweets;
private final Region<Long, Status> newtweets;

{...}

public TwitterCache() {
    cache = new CacheFactory().create();
    tweets = cache.getRegion("tweets");
    newtweets = cache.getRegion("newtweets");
}

{...}

}

Then, you will be able to access the cache and its “regions” just as a typical java HashMap. To store a value, you use the put(key, value) method and, to get a value, you use the typicall get(key) method. You may use any class to be your keys and values, but remember that classes should implement equals() and hashcode() methods to be used as keys on maps.

Now let’s assume we have a Gemfire cache full of tweets (using the example above), and we want to do a classical word count (like the wordcount example in Hadoop). Basically, we want to check the Top 20 words of our data set (excluding stop words). However, you don’t want to use hadoop because your data is in-memory and you want don’t want to move it to HDFS. Besides, you want a close to real-time view of what is happening on twitter. Then, you decided that you need an in-memory MapReduce. That could be hard to program some years ago, but with new features on languages such as Scala or Java 8, you can do it very easily. For example, for our in-memory map-reduce for tweets, we just need this method:


public static Map<String, Integer> wordcount(Map<Long, String> tweets) {
     StopwordFilter stopwords = new StopwordFilter();
     Map<String, Integer> result = tweets.values().stream().parallel()
         //convert each tweet in the Map<> tweets -> an array of words
         .map((tweet) -> tweet.trim().split(" "))
         //each array s is converted -> stream
         .map((s) -> Arrays.stream(s).parallel()
         //for each word, removes digits and other non-text chars
         .map(word -> word.replaceAll("[^#@\\p{L}\\p{Nd}]", ""))
         .filter(stopwords)
         //collect the words of the tweet and creates a map
         //map each word to it's counts
         .collect(Collectors
         .toMap(word -> word, word -> 1, (a, b) -> a + b)))
         //reduce all the resulting Maps into a single resulting map
         .reduce(new ReduceOperator()).get();

     return result;
}

Java 8 introduced a more functional-style programming. Thus, as you can see in the code above, we pass the stopwords object as a predicate to the filter method. The class StopwordFilter implements the Predicate<T> interface, which obligates the class to have the test(T value) method, that should return true whether the value attends a criteria. Check the class code below:


package com.tweet.cruncher;

import java.io.File;
import java.io.FileNotFoundException;
import java.util.HashSet;
import java.util.Scanner;
import java.util.Set;
import java.util.function.Predicate;

/**
* Functional predicate to check if a given word is a stopword.
* @author Jonas Dias
*/
class StopwordFilter implements Predicate<String> {

    Set<String> stopwords;
    public static final String STOPWORDS_EN = "stop_en.txt";
    public static final String STOPWORDS_PT = "stop_pt.txt";

    public StopwordFilter() {
        stopwords = new HashSet<>();
        try {
            this.loadStopWords(STOPWORDS_EN);
            this.loadStopWords(STOPWORDS_PT);
        } catch (FileNotFoundException fileNotFoundException) {
            System.out.println("Error while loading stopwords file. "
            + "Please check if you have the "
            + "files " + STOPWORDS_EN + " and " + STOPWORDS_PT
            + " in your running directory.");
            System.out.println(fileNotFoundException.getMessage());
        }
    }

    @Override
    public boolean test(String t) {
        return !stopwords.contains(t.toLowerCase().trim()) && !t.isEmpty() && !t.startsWith("http");
    }

    private void loadStopWords(String filename) throws FileNotFoundException {
        Scanner s = new Scanner(new File(filename));
        while(s.hasNextLine()) {
            String stopword = s.nextLine().toLowerCase().trim();
            stopwords.add(stopword);
        }
    }
}

Gemfire keeps data in-memory, thus it may be quite important to persist data to disk, eventually. Thus, I also wrote a simple class to lazy-write data to HAWQ. I use a PostgreSQL JDBC driver to connect with HAWQ and to perform inserts on the database periodically. Besides, a controller class manage the threads that collect data from twitter using Twitter4j. I am not showing the classes here, but you can find them in the zip file attached.

I found this example worth sharing because it shows how easy you can develop your Gemfire application. In order to run the Java app with the Gemfire cache, it is important to configure your cache. You can do this by placing a XML file on the same directory where you are running your Jar file. Here is the XML I have used.

<?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE cache PUBLIC "-//GemStone Systems, Inc.//GemFire Declarative Caching 7.0//EN" "http://www.gemstone.com/dtd/cache7_0.dtd">
<cache>
     <cache-server port="40404" />
     <region name="tweets">
         <region-attributes refid="REPLICATE" /> <!-- client region shortcut -->
     </region>
     <region name="newtweets">
         <region-attributes refid="REPLICATE" /> <!-- client region shortcut -->
     </region>
     <region name="wordcounts">
         <region-attributes refid="REPLICATE" /> <!-- client region shortcut -->
     </region>
</cache>

After running the cache for several minutes, you get several tweets on your cache. Besides you have the results for the wordcount. However, you need someway to visualize them. You may build a client app to access Gemfire cache and read the cache. I will leave the details for a next post! 🙂

You can also download Gemfire from Pivotal network and try this tutorial. The tutorial is good because it gives a good overview of the different types of caches and also gives an overview of the gfsh utility. More interested readers may also like this post from @william_markito.

The maven project with the code I used is available here.