It depends on whether we're talking about a first or second language. Children acquire their native language effortlessly, regardless of the language. Learning another language later on, however, is a different matter. Some languages do have far more complicated word-building rules than others, and others have far more complex sound patterns or sentence structures. But despite differences in individual areas of a language, researchers have not found any one language or group of languages to be clearly more difficult or complicated in all areas.
To some extent, how difficult it is to learn a language depends on how much it has in common with the language (or languages) that you already speak. Learning a language that is closely related to your native language can be easier than learning one that is very different. French, Italian, Spanish, and Portuguese are all descended from Latin, so they are closely related, and a speaker of one can learn any of the others fairly easily. Likewise, English, Dutch, and German are closely related, having all descended from an earlier language called Germanic, so it would be relatively easy for an English speaker to learn Dutch or German. But learning a language that's closely related to your native language can also bring problems, because their similarity can result in interference from your native language that would cause you to make mistakes.
A very different language such as Chinese, Turkish, or Mohawk, however, brings additional difficulties. In Chinese, for example, a 'word' is made up not just of consonants and vowels, but also the 'tone', or pitch, with which it is uttered. This means that the syllable ma uttered with a high tone ('mother' in Mandarin Chinese) is a completely different word from ma uttered with a low rising tone ('hemp'), which in turn is a completely different word from ma uttered with a high falling tone ('scold'). The words of such a language are likely to be very difficult for a native English speaker to master.
In short, no one language or group of languages can be said to be harder than the rest. All languages are easy for infants to learn; it's only those of us who grew up speaking something else that find them difficult.