James vented some of his exasperation during Saturday's blowout loss against the Toronto Raptors, a game the Cavs led 26-8 at one point.
They ultimately went down 110-93 to fall to a fourth straight defeat and move to a 5-7 record.
Afterwards, James pulled no punches in his assessment of the team's current mental toughness.
"We're a very fragile team right now. Well, we were a fragile team from the beginning," said the four-time NBA MVP.
"Any little adversity hits us, we just shell up. That's something that will come with experience."
Team-mate Kevin Love even suggested that they have allowed opponents to seize the initiative too easily in games.
"I think when we come out of a timeout and a team scores on us and gets a stop, scores on us again, we tend to definitely hang our heads," he said.
"I know that in certain aspects or in certain times, I'm guilty of it, too."
The Cavs were widely considered the standout team ahead of the start of the NBA season but have so far failed to deliver.
That said, amid the hype and artificial panic that gets generated around a team with so much star power and such high expectations, it is easy to forget how long it can take for a new team to come together.
The Miami Heat started 9-8 in 2010-11 before going on to make the NBA Finals, and James' experiences as part of that side have allowed him to keep the Cavs' slow start in perspective.
"This is not even the lowest it's going to get for us," he added.
"You guys know, the lowest it can get is up 17 in the fourth quarter of Game 2 of the Finals and losing.
"It can get that low. Or being up 2-1 and going to Game 4 and lose three straight in the Finals. That's very low. So, I'm very optimistic."