So what's up with this new review system?

[Midori]

Many of you may be wondering about that. It is very different from the old system; that much is clear. It is supposed to be intuitive to anyone who's used other web-2.0-like sites, but a bit of explanation may still be required for some of the nuances unique to our system. Therefore it may be a good idea to read these instructions before using the system.

Viewing the system
To view or submit ratings and reviews, go to the page for the title you want. For example, This page for K-ON!:
You will notice that there are several parts of the page:
1. The overall rating, at the top. This is an average of how much raters liked this title.
2. The summary, just below the overall rating. This is simply a bit of information about the show to give you an idea of what it is like. This is not a review!
3. The content bars, at the right. This approximates the amount of potentially objectionable content in this title, based on people's ratings. If one of these bars says 0 it does not mean that there is no objectionable content. It means that nobody has rated the title in that area.
4. The text reviews, which are at the bottom of the page. This area allows reviewers to give detailed thoughts on the title, or to explain the reasons for their ratings.

Rating reviews
Since there may potentially be a lot of text reviews for a series, you can rate the reviews, and your ratings will affect the order in which the reviews are displayed. Fortunately you don't have to assign numbers to the reviews; you just have to say whether you liked them or not. If you look at the upper-right corner of each text review, there will be links that say "+1" and "-1". Click the +1 if you thought the review was helpful and insightful. Click the -1 if you don't think the review is very important to read. These ratings will affect only the text reviews, and not the content ratings given by the same reviewer.

Submitting ratings
If you have an opinion you want to share with the world about a title, go ahead and submit a rating. It's super-easy! You don't have to be an English major. The only requirement is that you should have seen, read, or played the whole title before rating it (or at least most of it, if it's more than 100 episodes long). Since we can't enforce that, we have to trust that you'll be sensible in this regard. There is also a modest requirement of 25 posts before you can submit reviews. To rate a title, click the "Add your own rating" link on the page for that title.

The only part of a rating that is required is the overall rating--meaning how much you liked it. All the other parts are optional.
- You can give an overall rating without giving content ratings.
- You can give some content ratings without giving others (leave them as 0).
- You can give content ratings without giving a text review.
When you go to submit a rating, you will see some help text to guide how you rate things, and what your numbers will mean. It is important to read this while you are rating the title. Obviously there is much subjectivity regarding objectionable content, but if one member's 3 means the same thing as another member's 7, then our ratings will not be very useful. Keep in mind that 1 is the minimum of content in any category. A submission of 0 means you are not rating it in that category. Also, this should go without saying, but do not submit dishonest or exaggerated ratings.

When you are writing a text review, make it as long or short as you want. A small message explaining one of your numbers is fine, and a multi-paragraph review explaining how and why you liked/didn't like the series and going into detail about your ratings is also fine. In future revisions of our system, we may truncate extra-large reviews to a comfortable size on the main page, but there will be a link or button to expand the review to its full size.

You can also edit the ratings and reviews you have already submitted. When you are on the page for a title that you have already rated, the "add a rating/review" button will instead say "edit your rating/review" and when you click it you can change stuff, add or remove stuff; you can basically do anything you want to it, except delete it.

SPOILERS!
A frequent complaint about our old review system was that the reviews often contained a lot of spoilers. Now, a parent who is concerned with their child's well-being may want to know about every detail of risky content in a show; at the same time, an anime fan who is interested in watching the show may be very annoyed if the plot of the show is spoiled to them. Therefore in the textual reviews we ask that you use spoiler brackets when discussing a plot point of the title that could be a spoiler. The spoiler brackets do not work the same way as in the forums.
- [spoiler] tags: Unlike on the forum, the text in these tags will be rendered inline with the other text in your paragraph. Therefore it is best for small, single-sentence spoilers. The spoiler will appear when the viewer hovers their mouse over it. It'll look weird if the text inside it is too long.
- [longspoiler] tags: Use these tags for long, multi-line spoilers. It'll make a whole box for your spoiler.
- Incidentally, many other BBcode tags will work in the reviews too.

Suggesting a title
If you want to rate a title that's not in the database, or you see that a title that really should be there is not, then you can submit a suggestion for that title. All that requires is giving the English and Japanese names of the title, a brief summary of the title, and a few other pieces of info. These can be pulled off of just about any anime site. To avoid duplicate entries and other issues, title suggestions have to go through moderator approval, but this should go really fast, unlike our old system. Since you should see all of a series before rating it, we won't accept suggestions for a title if the title has not finished airing in Japan or been released yet.

Notice
We are taking this reviewing system seriously. If you attempt to abuse it, such as by submitting dishonest or deliberately misleading ratings or reviews, we will take action. We may revoke your ability to submit ratings, and we may give you a strike.

Anything I missed?
If you have questions or concerns, or if you don't understand anything, post in this thread about it! We'll try our best to answer you. Perhaps I'll modify this post and we'll build a little FAQ or something.

[MomentOfInertia]

If I post a general review (i.e. 7/10) cna I come in later and add ratings or a text review?

Or edit a review of an ongoing show after the current season ends?
Edit: Or for a horrible grammar error.

It says "the ratings system does not calculate a pure mean" are they weighted based on review count? post count? membership duration? distance from the median score? Or is it just a 'mods count double' set up?

I like how you used 'wanton' twice in the ratings descriptions. XD

[TopazRaven]

I really like the new review system. I actually tried it out myself a few hours ago and wrote a review for Axis Powers Hetalia. My first review for the site! I've never written a review before so hopefully it's alright, but I had a lot of fun. xD Anyway I don't have any questions so far, you guys did a great job with this and I just wanted to express both my thanks and approval. xD

[Midori]

If I post a general review (i.e. 7/10) cna I come in later and add ratings or a text review?

Or edit a review of an ongoing show after the current season ends?
Edit: Or for a horrible grammar error.

Yes, you can edit ratings/reviews you have already submitted. When you are on the page for a title that you have already rated, the "add a rating/review" button will instead say "edit your rating/review" and when you click it you can change stuff, add or remove stuff, basically do anything you want to it, except delete it. This is a good question, I'll add it to the OP.

It says "the ratings system does not calculate a pure mean" are they weighted based on review count? post count? membership duration? distance from the median score? Or is it just a 'mods count double' set up?

I'm not sure how the algorithm works exactly. You'll have to harangue Mithrandir about that.

I like how you used 'wanton' twice in the ratings descriptions. XD

About that too. : )

[MomentOfInertia]

Excellent.

[Mithrandir]

If I post a general review (i.e. 7/10) cna I come in later and add ratings or a text review?

Or edit a review of an ongoing show after the current season ends?
Edit: Or for a horrible grammar error.

It says "the ratings system does not calculate a pure mean" are they weighted based on review count? post count? membership duration? distance from the median score? Or is it just a 'mods count double' set up?

I like how you used 'wanton' twice in the ratings descriptions. XD

re: edit later: The "add your own review" link will be replaced with an "edit your review" link instead.

re: ratings: The function of the system includes a a fair bit of code that I'll be tweaking over time. It will include indicators regarding deviation from mean (Std Dev) and we'll probably do a bit of other calculations to try and ensure the data is statistically significant.


Edit: SNIPED by 6 minutes. XD

[MomentOfInertia]

Hmm interesting.

Thank you for the prompt response.

I'll help fill out the reviews now, the new format is much less intimidating than the old one, you can contribute with out committing to writing an essay.

[Midori]

Since it has become evident that it is required, I am adding a section in the OP warning against misbehavior in the reviews section.

[rocklobster]

is it ok if I have an essay anyway?

[goldenspines]

is it ok if I have an essay anyway?

This would be fine. But it's important to keep in mind who your audience is. Who is going to be reading the review you submit? Would it be beneficial for the review to be long or short? This will depend on the series and your rating, I would assume. Use your best judgement to help others.

And sometimes, some members will have more to say than others, but that doesn't make one review better over another one. ^_^

[rocklobster]

Thanks. I got so used to the old way, that it's kind of become my default mode.

[Yoshi]

I have never said I wanted to return to the old review system; despite what others have conjectured. The insinuations that have been made have taught me that the freedom of thought is not as alive here as I might have hoped...

If you actually want to know what I *REALLY* think about the review system, please PM me... I do *NOT* have an overtly negative opinion of it, contrary to popular belief.

However, I doubt anyone will message me, seeing as I doubt it would matter, even if they did...

Just remember: in the land of the blind, the one-eyed man is king!

Yoshi

[ABlipinTime]

@Yoshi
on point 2) The old system was where the mods would review the content reviews before they were displayed on the webpage. This made things censored, but it was really slow. That's why we have the new system.

I understand your points. There was a reason for the old system, obviously. It's just slow.

On your note about the majority of reviews will be from fans... That will probably be true, but my understanding is that a review with a dissenting opinion is not unwelcome. I already plan on reviewing certain shows that I don't approve the content of so that people can be aware of it. But hey, I'm just one little blip in time, right?

[TopazRaven]

I think the new review system is both fine and fair so I really don't know what you're going on about. This site is full of plenty of intelligent people, we aren't a bunch of idiots. I haven't seen anyone rate a series a 10 unless they truly thought it deserved such a rating. The new review system gives everyone a chance to not only write out their opinion, but has a system that rates each category. Sorry, but just because you think a certain title is horrid doesn't mean someone else does and everyone is entitled their own opinion. I think the mods are correct in the enforcing of not rating an anime unless you've actually seen the whole thing. Otherwise how can you give a proper review?

[Cognitive Gear]

In other words, your opinion needs to agree with the opinion of the people in charge, or else your data isn't statistically significant...

I guess I don't have the right to say this, so just ignore me... I predict that this new system will become about as useful as the one at Animeseason.com, or the the game-reviewing system at IGN--so if that's the type of thing you like, then great!:thumb:

My hypothesis is this: the majority of 'reviews' will be from FANS of the series (pretty obvious!) who will cause the entire ratings system to become inflated... I'm sure there are users who will rate multiple series a '10', even though that rating is supposed to only be reserved for the 'best'... and if you rate something a '1', you probably get your rating removed, don't you?

Without any lower ratings to compare the higher ratings to, the system will become worthless... of course, no one likes to hear this sort of thing, so why don't we just continue rating all of our favourite series '9' and '10' and forget about objectivity?

Well, I might as well mention two (out of many) possible fixes:

1. Allow each user to rate 10 series that they have seen at a time, giving them a ranking of between 1 and 10 -- this actually would even out eventually to a proper rating for each series, where the descriptions would actually fit! In other words, something rated a 5 would be 'worth my time, but probably not watching again' instead of 'worst show ever!'

2. Have at least TWO staff (moderators) who are familiar with the series review each CONTENT rating BEFORE it appears on the site... this may mean that the quality ratings are simply being thrown to the wind--if so, then so be it! However, the CONTENT ratings are what separates this Christian site from any other site... and they should be the most important!

Yoshi

DISCLAIMER: Take this nonsense at face value at your own risk! Due to current statuses, drastic measures are required to make a point--but seeing as the point being made is more important than the individual making the point, completely disregarding the individual is probably the best option...

Yeah, but honestly I would rather have protection against trolls that will rate popular series they dislike a '1', while also placing the content ratings up to '10'. That sort of thing generally indicates that they haven't seen the anime, or have some sort of moral objection to Christians watching the show, and I'd rather have reviews from people that have actually taken the time to watch the show.

EDIT: Also I think it's a good way to prevent the top 10 from becoming some kind of weird competition. Everyone likes to see their personal favorites on top ten lists, and sometimes people are a bit misguided and will attack "threats" to the spot. It's happened elsewhere on the internet, it can happen here.

[Radical Dreamer]

2. Have at least TWO staff (moderators) who are familiar with the series review each CONTENT rating BEFORE it appears on the site... this may mean that the quality ratings are simply being thrown to the wind--if so, then so be it! However, the CONTENT ratings are what separates this Christian site from any other site... and they should be the most important!

Speaking as a former mod, lolno. XD In the past, reviews had to be approved by one moderator, and it took forever. Keep in mind that the people who work on this site have lives outside of it, and watching all of the anime ever just so they can approve a review from one of the members is hardly efficient. Asking for two moderators to approve a review is just asking even more people to take time out of their very busy lives to watch an anime they otherwise wouldn't be interested in, just so they can approve someone's review about it. This system actually takes two pieces of the old system--the review page and the review thread, where other members would throw in their two cents on the accuracy/quality of the review or show in question--and merges them together with some newer features, allowing for a more streamlined system that's easier for everyone involved, and a lot more interactive to the user base. I see nothing but positives, here. XD

[Yoshi]

Apparently, you didn't read either of my suggestions, or my disclaimer, or you misunderstood both of them. I didn't say that a moderator should have to approve each review... nor did I say that I moderator should have to watch an entire series! I simply said that the CONTENT ratings should at least be looked at by moderators... but I understand how this is a moot point, from the looks of how things are run around here.

Furthermore, the obvious ratings inflating that has occurred/is occurring/will continue to occur is part of the cancer that is killing most review sources--and I doubt this site will fare much differently.

No matter, I guess I ought to take back my words, as I doubt that anyone else will care to see them!

Yoshi

[Mithrandir]

In other words, your opinion needs to agree with the opinion of the people in charge, or else your data isn't statistically significant...

You know, I really wasn't planning on taking action regarding you public. In fact, every time you've run afoul of the mods so far they've tried very hard to keep it private and have been pretty courteous. However, for reasons that make no sense to myself or my team, you seem intent on dragging your own issues across the board.

Very well - here's how this is going to go down: As tempted as I am to skip strike two and wash my hands of you, I'm only awarding you strike two.

One more, and you're off my board. Period. I'm mostly convinced you're just here to waste everyone's time]I simply said that the CONTENT ratings should at least be looked at by moderators... but I understand how this is a moot point, from the looks of how things are run around here.[/quote]

Listen, there is only one thing I expect from members here - treat everyone else with respect. You have mouthed off to at least one of my mods, which doesn't endear you to the staff. You rated Baccano! as a 1 overall and a 10 in every other content rating. The reviews system is meant to accurately reflect the content in the series, not allow you to distort the averages for your own amusement. Your actions on this board are showing very clearly that you are more interested in having fun than in contributing to the health of the board. I can accept that on one condition - you do so with respect.

If you're going to stay here, you're going to do so by adopting a different attitude. This is your final warning.

EDIT: And everyone else - I'm sure it goes without saying, but please stay out of this one.

[ABlipinTime]

:/
Yoshi:
I see what you mean about the content of reviews. As a drastic example, let's suppose someone cusses throughout their review. Whether there's cussing in the anime/manga/game the fact is that someone put something negative in their review and it got immediately posted to the main page. Not good for our site's reputation, so hopefully there aren't people like that here.

I understand numbers aren't a good way of rating something. Already I've been looking at trying to rate things, and the numbers just don't fit. Words are worth a thousand numbers (or is it "a picture's worth a thousand words?" XD) - there isn't a numeric substitute for just explaining the details. They give us a vague idea of what a handful of the people here on this website think - that's all. They aren't supposed to represent some absolute value of the anime itself. I see your point, and hopefully more people write reviews, not just throw out a few numbers.

EDIT: I see Mith beat me to responding.

[Cognitive Gear]

:/
I care to see what you've said, Yoshi.
I see what you mean about the content of reviews. As a drastic example, let's suppose someone cusses throughout their review. Whether there's cussing in the anime/manga/game the fact is that someone put something negative in their review and it got immediately posted to the main page. Not good for our site's reputation, so hopefully there aren't people like that here.

To my knowledge, CAA actually has a filter for swear words. I would test it, but I'm fairly certain that it would flag my post.

I understand numbers aren't a good way of rating something. Already I've been looking at trying to rate things, and the numbers just don't fit. Words are worth a thousand numbers (or is it "a picture's worth a thousand words?" XD) - there isn't a numeric substitute for just explaining the details. They give us a vague idea of what a handful of the people here on this website think - that's all. They aren't supposed to represent some absolute value of the anime itself. I see your point, and hopefully more people write reviews, not just throw out a few numbers.

Yeah, I definitely do think that number reviews don't represent things perfectly, but they do help me get a feel for what a reviewer thinks of something at a glance. In our case here, I think it also offers some interesting insight on what kinds of shows the community likes overall, so I hope that everyone rates all of the shows that they have seen.

[blkmage]

Numbers are also useful in aggregate. With a large enough sample size, even with outliers, it's possible to gauge what a community thinks of a particular show. Something I've found that was pretty neat on MAL was that they gave you graphs of ratings for each of their shows.



Even with the exact numbers and percentages removed, you can see that the ratings follow the same distribution, but simply have their mean shifted. Obviously, this can't tell you whether you'll like a show or not, but it is possible to derive some meaningful information from these stats.

The normal distribution is one of the reasons why your standard 10 point scale still works and why adjusting things based on standard deviations and other statistical magic is good enough to account for outliers. And when reading general trends like this, score inflation isn't an issue, because it's really obvious to account for.

Anyhow, the objections regarding score inflation seems to be a misunderstanding of what scores are supposed to represent and a misguided expectation that they're supposed to act as deciles or something.

P.S.: if you were wondering what the graphs are for, it's The Melancholy of Haruhi Suzumiya, The Melancholy of Haruhi Suzumiya (2009), and The Disappearance of Haruhi Suzumiya. Notice the huge drop between the first and second season and the comparatively astronomical ratings for the movie.

P.P.S.: I think it'd be awesome if we the spoiler tags used in the reviews were used across the forums, because I have trouble reading the highlighted text on my setup.

[ABlipinTime]

Numbers tell you something up to a point. But the numbers in those graphs are rather meaningless. What are they rating? What does "10" mean? Does it say how much someone likes it? Possible rewatcher? We're also not considering here the phrasing that is used to tell someone which numbers to select (if there is such phrasing), such as in our system: e.g. Give this a 2 if it shows cartoon violence, a 3 for blood. My point being, the numbers can tell you things to a point. It's not the outliers that make the numbers meaningless to an extent - it's the fact that there isn't really a scale. Consider ANN's scale. Based on experience, I had to actually watch shows to really understand that things below 7.24 in overall rating would probably not interest me. Yeah, that gives me some idea of whether it's good, so yeah, the number is worth something, but only to a point.
Sorry for my verbose redundancy. I kept thinking I hadn't said things the way I wanted.

[blkmage]

Like I said, numbers are for judging opinions of a show in aggregate, rather than from a specific individual. All of those questions you're asking regarding specificity become statistically irrelevant as the number of ratings grows, which was my point. What the graphs and numbers tell you is the overall sentiment of a group of people and how that sentiment trends.

Will that alone tell you whether or not you should watch something? No, of course not. Is that information useful when you use it in conjunction with other pieces of information? Yes, and that is what I'm saying its value is. You have to know the proper context in which you're reading numbers in order to make them useful.

[Cognitive Gear]

Like I said, numbers are for judging opinions of a show in aggregate, rather than from a specific individual. All of those questions you're asking regarding specificity become statistically irrelevant as the number of ratings grows, which was my point. What the graphs and numbers tell you is the overall sentiment of a group of people and how that sentiment trends.

This is true, but I don't think that CAA is large enough (in terms of active users that will rate titles) to not take the pre-emptive measures just yet. Otherwise, I would prefer that we be rid of such a thing, as it would be a more honest measure of the community's feelings.

EDIT: Also if someone would be so kind as to let me know if I can watch the Haruhi movie without having seen the series, I would appreciate it.

[ABlipinTime]

I don't see how my questions grow irrelevant.

(On second though, I also don't see why I'm in a pointless debate at 11:44 at night. Gee, I'm funny... or just tired. And now I'm rambling.)

[rocklobster]

Would it be a bad idea to do a review for a title that was reviewed on the old list?

[Midori]

Would it be a bad idea to do a review for a title that was reviewed on the old list?

It would in fact be a good idea. The old list is still there in case there are links to it, but the new system is intended to entirely replace the old system. If a title in our old list is not in our new database yet, please submit a suggestion for it.

However, if you're planning on copying an old review of yours into the new system, look it over real well to see if it still fits into the new system, because our content categories are slightly different. And also be sure to employ spoiler brackets to hide any spoilers that may have been in the old review. Also, don't copy over an old review if somebody else wrote it. Let them do it, or get their permission.

We considered automatically migrating all our old reviews to the new system. However, the systems were deemed too incompatible to copy reviews except on a case-by-case basis.

[Kaori]

When you go to submit a rating, you will see some help text to guide how you rate things, and what your numbers will mean. It is important to read this while you are rating the title.

Excellent. I was just looking at the old review database not long ago and thinking that something like this would be really helpful.

Personally, I think it would be more helpful to see the complete list of titles as the top page rather than just a list of the top 10, but that’s just my own preference. Other than that, the more I look through the new system, the more I like it. The organization is great (quick statistics at the top, followed by a short summary, then specific individuals’ comments), and as more people add reviews, it could potentially become a very helpful resource; it will be nice to be able to see multiple people’s comments on the series all in one readily accessible place.

Oh, and those spoiler tags are pretty spiffy, too. ^^

[Edward]

I really like the new review system, from what I've seen of it. And I'm glad that the old reviews are still available. I've bookmarked it just in case I want to read a review for something that isn't in the new system. My favorite thing about the new system, though, is that you can rate titles without writing a review. The aggregate rating system gives a much more objective rating than the previous system. So thanks guys!

[rocklobster]

Just tried things out! Here's your first test!